Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Set Up an AI Operator for a ServiceBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Add an AI Operator to a running Service to measure what it costs and how well it answers, then ask it for cheaper configurations.

About This Task​

Before you start, make sure you have:

  • A running Service you want to optimize.
  • An Experiment for that Service's pipeline, with evaluators that judge answer quality. If the pipeline has no Experiment yet, the Add AI Operator modal tells you so and links to where you can create one.

Each Service can have only one AI Operator. If a Service already has a goal, it still appears in the Add AI Operator list, but it's disabled.

Add an AI Operator​

  1. Go to Operate > AI Operator, or open a Service and click Set up AI Operator next to the actions menu.
  2. Click Add AI Operator.
  3. Select the running Service you want to optimize.
  4. Select the Experiment whose evaluators judge this Service's answers.
  5. Under Goal, select what to Optimize for. Optional: Set Min Quality score (0 to 1). A proposed configuration must always match the quality of the Service's current configuration. If you set a minimum, it must reach the current quality or your minimum, whichever is higher. Hover over the ? icon next to a label for details.
  6. Click Save to create the goal.

Result: The Service now has an AI Operator. Its row appears on the Operator list with a status based on its latest run, and the Service page shows an AI Operator button in place of Set up AI Operator.

View the Operator List​

To check which Services are being optimized, whether each has a baseline, and when each was last measured, go to Operate > AI Operator. The list shows every optimization goal in the workspace. For each row, you see:

  • Service: the Service the goal belongs to.
  • Goal: the goal written as a sentence, so you can read what it keeps and what it optimizes for.
  • Status: read from the latest measurement run (for example, whether a baseline exists or a run is in progress).
  • Last run: when the Operator last measured the Service.

Click a row to open that Operator's page.

Measure a Baseline​

Measure a baseline to record the Service's current cost, latency, and quality. You need a baseline before you can request suggestions, and it gives you numbers to compare them against.

  1. Open the Operator page for the Service, either from the Operator list or the Service page.
  2. Choose the pipeline version you want to measure.
  3. Start the measurement run. The Operator page switches to the new run so you can follow its progress.
  4. Review the baseline once the run finishes: cost, latency, and quality, along with the conversations the run used.

View Past Runs​

To compare what the Operator measured and suggested over time, or to look back at an earlier run, use the Runs table on the Operator page. The table lists every run of the Service's Operator, newest first. It appears once the Service has two or more runs.

For each run, the table shows:

  • Started: when the run started. Hover over the time to see the exact date. The newest run has a Latest tag, and the run currently shown on the page has a Viewing tag.
  • Version: the pipeline version the run measured. If the version isn't known, the table shows a placeholder instead.
  • Conversations: which conversations the run measured: sampled, hand-picked, or from a dataset.
  • Tested: how many configurations the run tested out of the maximum, for example "2 of 3 configurations".
  • Result: how the run ended. One of Running, Found an improvement, No improvement, Inconclusive, Stopped, or Failed.

The table shows 10 runs at first. To load 10 more, click Show older runs.

You can open an earlier run by clicking it in the Runs table.

Fix a Refused Run​

An AI Operator run can be refused when some of the Sessions it would use can't be replayed, or when too few Sessions remain. The error message names each affected Session by its name and its id, in the format '<name>' (<session id>). Several Sessions can share a name because a name is usually the Session's first query. Use the id to tell them apart.

Typical reasons for a refusal include:

  • A Session has turns that recorded no inputs because content tracing was off, so its requests can't be replayed.
  • A Session was recorded on a different pipeline from the one the Service runs.
  • Too many Sessions failed when the baseline was replayed on them.
  • Too few Sessions are left to hold out some for judging a candidate. This can happen when recent traffic is used and Sessions are dropped because they recorded no inputs.

To fix a refused run:

  1. Read the error message and note the Session ids it lists.
  2. On the Operator page, check the run's Left out list. The Session column names each Session, and the Why column gives the reason it was left out.
  3. Remove the Session from the Dataset, or fix the cause (for example, turn on content tracing for the Service so new Sessions record their inputs).
  4. Start the run again. If the run is refused again, repeat these steps for the Sessions the new error message lists.

If you start runs through the API, a refused Dataset run returns a 422 response. Besides errors and test_case_ids, the response lists the affected Sessions in session_ids. Other refusals, such as too many Sessions that couldn't be replayed or too few Sessions left, name each left-out Session by name and id in the error message only. Use these ids to find the Sessions and their traces. To learn where to view traces, see Trace with Built-In Traces.

Review Suggestions​

Once a baseline exists, you can ask the AI Operator for optimization suggestions:

  1. On the Operator page, request suggestions.
  2. Review each suggested configuration together with its validated cost and quality.
  3. Copy a suggestion's configuration if you want to try it yourself. The AI Operator does not apply or promote any suggestion automatically — you decide what to do with it.

To act on a suggestion from the latest run, click Accept to roll out its configuration, or click Decline to dismiss it. You can accept or decline suggestions only for the latest run. Suggestions from earlier runs are read-only.

Edit or Remove the Goal​

On the Operator page, you can:

  • Edit the goal to change what to Optimize for or the Min Quality score. You can't change the Experiment after you create the goal. Saving a goal that previously allowed some quality loss resets that allowance to zero, so proposed configurations must match the baseline's quality.
  • Remove the goal, which also removes the Service from the Operator list.

What To Do Next​