AI OperatorBeta
An AI Operator is an optimization goal set on one Service. Use it when you want to lower what a Service costs without lowering the quality of its answers. It measures what the Service costs and how well it answers, then suggests cheaper configurations for you to review.
What an AI Operator Does
Each Service can have at most one AI Operator. When you add one, you choose:
- The running Service you want to optimize.
- The Experiment whose evaluators judge the quality of that Service's answers.
- The quality the Service must keep while being optimized.
The AI Operator then measures a baseline for the Service: its cost, latency, and quality on the pipeline version you choose, based on a set of conversations. Once it has a baseline, you can ask it for optimization suggestions. The AI Operator proposes alternative, usually cheaper, pipeline configurations and validates that they still meet the quality you set.
Suggestions are advisory only. The AI Operator does not apply or promote configurations by itself — you review what it finds and decide what to do with it.
Where You Find It
The AI Operator has two entry points:
- Operator list (go to Operate > AI Operator): to check on all optimization goals in the workspace, use this list. It shows each goal's Service, the goal written as a sentence, a status read from the latest measurement run, and when it last ran. Click a row to open that Operator.
- Service page: to set up or open the AI Operator for a specific Service, use the Service page. A Service with no goal shows a Set up AI Operator button next to the actions menu. A Service that already has a goal shows an AI Operator button instead, which opens its Operator page directly.
Key Ideas
- One goal per Service: you can't add a second AI Operator to a Service that already has one. The existing Service still appears in the Add AI Operator list, but it's disabled.
- Experiment-backed quality: the quality threshold the Operator must keep comes from the evaluators defined in an Experiment. A pipeline with no Experiment can't get an AI Operator until you create one.
- No quality loss: a proposed configuration must match the quality of the current configuration (the baseline). To require a higher quality, set a Min Quality score on the goal.
- Fair quality comparisons: some evaluators answer not applicable on conversations where their question doesn't apply. To compare like with like, the AI Operator averages each score metric of the baseline and a suggestion over only the conversations both of them scored. It then checks the quality floor against those averages. So a good suggestion isn't rejected just because the baseline and the suggestion scored different conversations. Runs recorded before scores were kept per conversation are compared as before.To get fresh scores, see Review and Re-Run Optimization Evaluations.
- Baseline before suggestions: before the AI Operator can propose anything, it needs a baseline measurement run on a chosen pipeline version using a set of conversations, split between training and held-out sets. The optimizer reads the training half before it suggests changes. The held-out half is kept back to measure the baseline and every suggestion, so scores on it come from conversations the optimizer never saw.In the baseline's evaluation, each conversation is labeled Training or Held out.
- Small runs are allowed: a run can start from as few as 9 conversations (Sessions), so you can start optimizing a Service before it has much traffic. The AI Operator splits them 70/30: the larger share is shown to the optimizer, and the smaller share is held out to judge each suggestion. With 9 conversations, each suggestion is judged on 3 held-out conversations, so its verdict is less reliable than on a larger run. Runs with fewer than 50 conversations are marked as a small sample.
- Every run is kept: the Operator keeps each run and its suggestions. The latest run is the one the Operator is working on now. You can open an earlier run to see what it measured and suggested, but only the latest run's suggestions can be accepted or declined.
How the Baseline Is Replayed on Test Cases
When a measurement run uses test cases from a Dataset, the AI Operator measures the baseline on each test case's recorded session if that recording can stand for the baseline. Otherwise, it replays the baseline on the test case and measures the replay. A replay is a full run of your pipeline for every turn, so it takes time and uses tokens. To avoid replays, choose as the baseline the pipeline version that answered your recorded test cases. The AI Operator replays only when it has to.
The AI Operator replays the baseline on a test case for one of these reasons:
OTHER_VERSION: at least one turn of the recorded session was answered by a pipeline version other than the baseline's version. A later turn depends on the earlier ones, so the AI Operator replays the whole session and never reuses part of it.NO_VERSION: at least one turn of the recorded session has no pipeline version. Nothing says which configuration answered it, so the AI Operator replays it.
If every turn was answered by the baseline's version, the AI Operator measures the test case as recorded and doesn't replay it.
A setup script on a test case doesn't force a replay. A session recorded on the baseline's version already ran in the state that the script recreates. When the AI Operator tests candidate configurations, it still runs the setup script before each replay.
See What a Replay Produced
To compare a test case as it was recorded with the way the baseline answered it, and to see why the AI Operator replayed it, open the replayed sessions from the measurement run's progress or list them through the API. For steps and the fields the API returns, see Set Up an AI Operator for a Service.
baseline_session_id: the session the baseline produced when replayed on the test case. It's null until the replay happens.baseline_query_ids: the turns of that session, in replay order. It's empty until the baseline has been replayed on the test case.replay_reason: why the baseline is replayed, eitherOTHER_VERSIONorNO_VERSION. It's null when the test case is measured as recorded, and for runs frozen before this field was kept.
Use these fields to compare a test case as it was recorded with the way the baseline answered it, and to see why the AI Operator replayed it.
Related Information
Was this page helpful?