Evaluation ExperimentsBeta
An experiment is how you answer the same quality question repeatedly: you run a set of metrics over your pipeline's sessions, change your pipeline, and run it again. To compare two pipeline versions fairly, you run them on the same test cases.
How an Experiment Run Works
An experiment is a named set of metrics you run over your pipeline's sessions. For example, an experiment called "Groundedness and cost" could combine an LLM judge for groundedness with a code metric for token usage. To keep related metrics together and reuse them, group them in a Metric Group. For details, see Evaluation Datasets and Metric Groups.
Like metrics, experiments belong to your workspace, not to a single pipeline. You define the set of metrics once and use it to judge any pipeline in the workspace. Each run records which pipeline it judged.
Each time you run an experiment, you pick the sessions to measure. The run gives you a scorecard for each metric, with its distribution and any sessions worth a closer look, such as where the metric broke, a gap between sources, or a rare answer. From a scorecard, you can inspect the sessions behind it and see each turn and its rationale.
Comparing Runs
Two runs are comparable only if they measure similar sessions. If one run covers simple questions and the other covers hard ones, the difference in scores reflects the questions, not your pipeline. To compare two pipeline versions fairly, both need to answer the same or similar queries. The simplest way to guarantee that is a Candidate and Baseline Experiment.
Candidate and Baseline Experiments
When you want to know whether a new pipeline version beats the current one, compare them on exactly the same questions instead of comparing two separate runs. Create an Experiment that sets a Candidate against a Baseline. Using the same test cases on both sides means any difference in scores comes from the pipeline, not from the questions.
An Experiment of this kind combines four things:
- Dataset: The test cases both sides answer. A test case is one recorded conversation, together with optional setup and cleanup scripts that run around every replay of it.
- Metric Group: The set of metrics that scores them. Both sides use the same one, so the results are comparable.
- Candidate: The pipeline version you want to evaluate.
- Baseline: What you compare the Candidate against: another version of the same pipeline, or the Dataset's recorded conversations.
You also choose repetitions per test case: how many conversations each side scores for each test case. Several repetitions help you see how much a pipeline's answers vary from one run to the next.
Replays
A replay sends a test case's requests to a pipeline version again to produce a new conversation. You choose whether a replay sends only the first message or the whole conversation.
Replay With Automatic Approval
Some pipelines pause and ask for approval before they continue, for example before an agent runs a tool or takes an action. A replay that sends only the first message stops at that question, so the conversation you score is shorter than the real one. To test the conversation up to and including the approval, use the First message, then auto-approve replay mode (FIRST_USER_MESSAGE_AUTO_APPROVE in the API).
In this mode, the platform replays the first request of the test case. After each reply, a replay guide decides what happens next. The replay guide is an LLM pipeline that plays the user:
- If the pipeline asks for something to be approved, the guide writes an approval. The platform sends it as the next request in the same replay conversation.
- If there is nothing to approve, or the pipeline only suggests a follow-up that isn't a direct continuation of the original request, the replay ends.
The guide sees the replay so far. It also sees the messages the real user sent later in the original conversation, as a hint of what that user approved. A replay sends at most five approvals by default. If the replay guide fails, the replay fails, so you never score a shorter conversation without knowing it.
Set Up the Replay Guide
The replay guide runs inside your deepset instance, on the model credentials of a dedicated workspace. The conversation content stays on the instance, and your own model provider isn't billed for it. Until an administrator configures the guide, requests for the auto-approve replay mode return a 422 error. This applies to a single replay, a Replay Suite, and a comparison.
To configure the guide for an instance:
- Create a dedicated workspace for the guide.
- In that workspace, create a pipeline named
Session-Replay-Guide(or another name that matches the setting below). Upload its YAML, mark a version as the default, and add the model provider secrets the pipeline needs. - Set the instance settings in the following table.
| Setting | Default | Description |
|---|---|---|
SESSION_REPLAY_GUIDE_WORKSPACE_ID | None | ID of the workspace that hosts the guide pipeline. The auto-approve mode is unavailable until you set it. |
SESSION_REPLAY_GUIDE_PIPELINE_NAME | Session-Replay-Guide | Name of the guide pipeline in that workspace. |
SESSION_REPLAY_GUIDE_TIMEOUT_SECONDS | 60 | How long the platform waits for the guide to answer. |
SESSION_REPLAY_AUTO_APPROVE_MAX_FOLLOW_UPS | 5 | Maximum number of approvals sent in one replay. |
The pipeline's inputs mapping can declare any of these inputs:
conversation: The replay so far, as a list of messages withroleandcontent.source_user_messages: The messages the original user sent after the first message.query: An instruction to decide whether to approve.messages: Sent empty. An Agent builds its user message from its own prompt.
The pipeline must return a JSON object somewhere in its output. To approve, return {"decision": "APPROVE", "message": "<the approval to send>"}. To end the replay, return {"decision": "END", "reason": "<why>"}.
To tell whether your results are ready, check the Experiment's status: Replaying while conversations are replayed, Scoring while the Metric Group is applied, and Finished or Failed.
Results
The Experiment is finished when both sides are scored. The results show the Candidate's value for each metric, the change against the Baseline, and the number of test cases scored on both sides. You can also ask the assistant for an AI Summary of how the Candidate compares to the Baseline. For steps, see Get an AI Summary of an Experiment.
To help you tell "didn't apply" apart from "errored" or "not scored yet", the results show, for each side, metric, and test case, how many results a metric declared not applicable. A test case where a metric returned only not-applicable results still appears in the results with that count.
A label metric has no mean, so its change against the Baseline is how each label's count and share of the side's labels moved, for example pass +3 (+43%). Results group metrics by the Evaluator that produced them, and every test case lists every metric of the Metric Group. A metric with no value for a test case shows —. For how to read these results, see Read the Results Table.
To start an Experiment, see Compare Two Pipeline Versions.
Label Changes Against the Baseline
The results chart how each label's share of the test cases moved against the Baseline: one bar per label, growing right in blue when the label's share grew and left in orange when it shrank. A label growing isn't good or bad on its own: more grounded answers is an improvement, more not grounded answers isn't. For how to read the chart, see Label Metrics.
Built-In Pipeline Metrics
Besides the metrics in your Metric Group, the results table and the test-case table include built-in rows that show what the pipeline spent to produce each session. To see whether a Candidate that scores better also costs more or runs slower, read these rows next to your quality metrics:
- Duration: the pipeline's run time in seconds, summed over a session's turns.
- Tokens: the tokens the pipeline's LLM calls used, shown as input and output together on one line, for example
1K in / 200 out. In is the tokens sent to the LLMs, such as the prompt and retrieved documents. Out is the tokens the LLMs generated. The Δ vs Baseline column shows the change for each part, for example+200 in (+25.0%) / +100 out (+100.0%). Comparing the two parts shows whether a cost change comes from a longer prompt or a longer answer. - Cost: what the LLM calls cost in USD. A session counts only if every call in it is priced, so unpriced sessions don't understate the spend.
Each value is the average over the sessions of a test case, and the change against the Baseline counts only test cases that have a run on both sides. To find which test cases make the Candidate cost more or run slower, read these values for each test case. For how to read the rows, see Read the Results Table.
Related Information
Was this page helpful?