Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Run an ExperimentBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

An experiment measures your pipeline with a set of metrics on the sessions you choose. Use it to see how your pipeline performs overall, find the conversations where it struggles, and check whether a change made it better.

About This Task​

An experiment is a question you come back to. You create it once, with the metrics that matter for your use case, and run it whenever you want an up-to-date answer: after you change a prompt, switch models, or when new production traffic comes in.

An experiment over sessions doesn't run your pipeline again. It measures sessions your pipeline already had. To measure a change this way, send queries to the changed pipeline first, and then run the experiment on those sessions. To have the platform replay test cases on two pipeline versions and compare them, see Compare Two Pipeline Versions.

For background, see Evaluation Experiments and Evaluation Metrics.

Prerequisites​

  • You have at least one metric you've tested and trust. To create a metric, see Write and Test an Evaluation Metric.
  • Your pipeline has sessions to measure. Every query sent to the pipeline, from Playground, a deployed app, or the API, creates one.

Create an Experiment​

  1. In the sidebar, under Testing, click Evaluation. To see only one pipeline's data, open that pipeline and click Evaluation instead.
  2. Go to Experiments>New experiment.
  1. Give your experiment a name that describes what the experiment answers, for example Groundedness and cost.
  2. In Metrics, choose the metrics to run. Pick a small set that covers what matters most, for example one metric for answer quality and one for cost. The order you pick them in is the order of the columns in the results.

Experiments always use the latest version of each metric. When you improve a metric, the next run picks up the change, and past runs keep their original results.

The set of metrics you save is a Metric Group. It belongs to your workspace, not to this pipeline, so you can run the same Metric Group on any pipeline and compare the results. Before you create a new experiment, check whether a Metric Group with the metrics you need already exists.

Run the Experiment​

  1. Open the experiment and click Run experiment.
  2. In Sessions to judge, choose the sessions to measure. Choose sessions that represent what you want to learn about. To see how the pipeline does for real users, pick recent production sessions. To check a change, pick the sessions you created after the change.
  3. If the experiment has already measured some of these sessions, choose what to do under Sessions this experiment has already judged:
    • Fill in the gaps only: Measure only sessions without a result. Use this to add new sessions to the picture without repeating work.
    • Judge everything again: Measure all selected sessions. Use this after you change a metric and want every session measured with the new version.
  4. Check the number of results the run produces. It's the number of sessions times the number of metric results. LLM judges call a model for each result, so a large run takes longer and costs more.
  5. Click Run.

A run over hand-picked sessions can measure up to 100 sessions and produce up to 500 results. Your workspace can have up to three runs in progress at once.

Run the Experiment Over a Time Window with the API​

To measure your live traffic without picking sessions one by one, use the API to run the experiment on every session in a time window, for example the last seven days.

A window contains every session of the pipeline whose last activity falls between from and to. You can narrow it with the same filters the session list uses:

FieldDescription
fromRequired. Start of the window, inclusive. Must include a timezone.
toRequired. End of the window, inclusive. Must include a timezone, be later than from, and not be in the future.
originOptional. Only sessions from this origin.
deployment_idOptional. Only sessions served by this service.
pipeline_version_idOptional. Only sessions served by this pipeline version.
revision_idOptional. Only sessions served by this service revision.

The window is always absolute. If your app offers presets such as Last 7 days, convert them to exact from and to times before you send the request.

Preview the Window​

{
  "from": "2026-10-01T00:00:00Z",
  "to": "2026-10-08T00:00:00Z",
  "origin": null
}

The response contains:

  • sessions: The number of sessions in the window.
  • more_than: If true, the window holds more sessions than the ceiling allows, and a run over this window is refused. Narrow the window or add a filter.
  • evaluators: The number of metrics in the experiment.
  • cells: Sessions times metrics, which is the number of results the run produces.
  • ceiling: The most sessions (sessions) and results (cells) a window run can have.

The preview count is an estimate. Sessions that arrive before you start the run can change it.

Start the Run​

To start the run, send the window in the window field when you create an experiment run.

Set exactly one of session_ids and window:

{
  "window": {
    "from": "2026-10-01T00:00:00Z",
    "to": "2026-10-08T00:00:00Z"
  },
  "reuse": "REUSE_SUCCEEDED"
}

The run measures the sessions in the window when the API receives the request. Traffic that arrives later isn't included. reuse works as for hand-picked runs.

The run is kept in the experiment's history. When it finishes, read its results as described in Read the Results.

Window runs have these rules:

  • A window run is always a sample of your traffic, so you can't set selection to HAND_PICKED.
  • You can't use focus with a window, because the sessions don't exist until the API resolves them.
  • The run keeps its window and the number of sessions it was launched with (resolved_sessions). Both appear when you read the run.

Window Run Limits​

A window run can measure up to 500 sessions and produce up to 1,500 results. These limits are separate from the limits for hand-picked sessions. A large window run can take a while to finish, because it shares the evaluation queue with other runs. For how the queue works, see Runs Waiting for Capacity and Transient Failures.

The API returns a 422 error in these cases:

  • The window has no sessions.
  • The window is over the limits. The error detail contains a message, the count of sessions, and the ceiling, in the same shape as the preview response. Narrow the window or add a filter, and try again.

Runs Waiting for Capacity and Transient Failures​

When you start many runs at once, for example a large experiment or a comparison, your organization's runtime and your judge's model provider can only handle so many calls at the same time. To keep runs from failing under that load, the platform limits how many Evaluation Runs of one organization execute at once. By default, up to 10 run at the same time. The rest wait in the queue and start as earlier runs finish.

A run that waits for capacity stays in the Created state. The wait doesn't count against the run's time budget. If a run still hasn't started after 6 hours, the platform marks it as failed with a capacity error. The experiment run still closes, so the results you have remain available.

If a call fails for a temporary reason, the platform retries it automatically, up to three attempts in total and within the run's time budget. It retries when:

  • The runtime has no worker free, is unavailable, or asks the platform to slow down.
  • The model provider that powers an LLM judge rate limits the request, is overloaded, or times out.

The platform doesn't retry a metric whose code is broken, because it fails the same way every time. If all attempts fail, the cell shows the last error, as described in Read the Results.

A retry calls your LLM judge again, so it can add to your model provider costs. The limit of three attempts keeps this cost bounded.

Read the Results​

When the run finishes, each metric gets its own scorecard:

  • For a Score, the scorecard shows the mean and the p10 — the value below which the worst 10% of sessions fall — so you see the typical answer and how bad the worst ones get.
  • For a Label, the scorecard shows the most common label.

A scorecard also flags anything worth a closer look: how many sessions broke because the metric's code raised an error or the run failed, a source gap between where sessions came from, such as different Services or pipeline versions, any rare answers for a Label metric, and how many sessions are N/A because the metric didn't apply.

Click a scorecard to open its breakdown:

  • Distribution: the same chart as before, paired with a table that breaks the mean down by source, so you can tell whether a difference comes from the questions or from the pipeline that answered them. Click a source row to inspect the sessions behind it.
  • Evidence: the sessions most worth a look for this metric. It shows the sessions where the metric broke; if none broke, it shows the rarest labels for a Label metric or the lowest-scoring sessions for a Score metric. Click a row to inspect that session.
  • Click Inspect all Sessions to open every session this metric measured.

Any of these opens the session inspector: pick a session, then a turn, to see the value and the rationale behind it, along with the trace. A session whose traces are no longer available shows that it has no traces instead.

For metrics that answer per turn, select how to roll up each conversation: Mean per conversation for the typical answer, or Worst turn per conversation to find conversations where a single answer went wrong.

Sessions where a metric doesn't apply are counted as N/A and left out of the distribution.

Compare Runs​

Each run is kept in the experiment's history. Choose a run from the run picker to see its results, along with who started it, when, and which metric versions it used.

To compare two runs, make sure they measured similar sessions. If one run covers easy questions and the other covers hard ones, the difference comes from the questions, not your pipeline. To compare two versions of your pipeline fairly:

  1. Run the same set of queries on each version.
  2. Run the experiment once on the sessions from each version.
  3. Compare the distributions and look at sessions that changed.

To compare two versions on the same test cases without sending queries by hand, see Compare Two Pipeline Versions.

Evaluate a Single Turn From a Trace​

When you spot a problem while inspecting a trace, you can measure the whole conversation it belongs to on the spot:

  1. Open your pipeline and click Analytics > Traces, then open a trace.
  2. Click Evaluate this conversation.
  3. In Choose an Experiment, choose the experiment to run, and click Run on this conversation. If the pipeline has no experiments yet, click Go to Experiments to create one.
  4. Click Open in Experiment to see the result with its rationale.

The run judges every turn of the conversation. Runs started with Evaluate this conversation on a trace appear in the Experiment's run history tagged hand-picked. They don't feed the Experiment's trends. When you follow an Open in Experiment link, the Experiment opens on that run instead of the latest one. To learn more about working with traces, see Evaluate a Conversation from a Trace.

What To Do Next​