Evaluation Datasets and Metric GroupsBeta
A Dataset holds the test cases you measure your pipelines on, and a Metric Group holds the metrics you score them with. Together they are the inputs of an Experiment that compares two pipeline versions.
Datasets
A Dataset is a collection of test cases that you keep for measuring your pipelines. Datasets belong to your workspace, not to a single pipeline, so one Dataset can hold test cases from several pipelines. Use a Dataset to keep the queries that matter to you, for example questions your users often ask or conversations where the pipeline failed, and reuse them whenever you measure a change.
Each test case shows the pipeline it comes from, so you can tell test cases apart when a Dataset mixes pipelines. To work with the test cases of one pipeline only, open the Datasets pages on that pipeline's Evaluation tab. They show only that pipeline's test cases.
You can build a Dataset in these ways. For steps, see Build an Evaluation Dataset.
- Add from Search History: In Search History, select Add to Dataset on a query. The test case keeps a reference to the pipeline the query ran on.
- Copy from another Dataset: Copy test cases from any Dataset in the workspace into the Dataset you're working on. Use this to reuse a good set of test cases instead of building it again.
- Import from an archive: Import a Dataset from an archive. Each imported conversation is stored with the Dataset under a new Session ID, so search history doesn't hold it. You can still read it and open the trace of each turn.
- Add from a table: Upload a CSV or XLSX file and turn each row into a test case. Use this when you have questions and expected answers in a spreadsheet but no recorded conversations.
To see how many Datasets, metrics, and Experiments your workspace has, check the tiles on the Evaluation Overview page.
You can export a Dataset as a .zip archive and import it into another workspace, for example to run the same regression checks on a staging pipeline. The archive holds the conversations themselves, so imported test cases replay like any other test case even though their conversations aren't in search history. For steps, see Import and Export a Dataset.
Table-Based Test Cases
A table-based test case is one row of a CSV or XLSX file instead of a recorded conversation. Use it to measure a pipeline on questions you wrote down, for example a spreadsheet of questions with the answers you expect. Each column of the file has one of two roles:
- Input: A value sent to the pipeline, for example
query. - Expected: Data the evaluator reads to judge the answer, for example
expected_answer. The pipeline never receives it.
A test case that replays a recorded conversation can carry expected data too.
Table-based test cases work like this:
- A row is named after its
queryinput. If the row has noquery, it is named after its first text input. - A replay of a row sends one request. The
queryinput is sent as the query, and every other input is passed to the pipeline input with the same name. Inputs returned by a test case's setup script take precedence over the row's inputs. - Because a row has no recorded conversation, the Whole conversation replay option doesn't apply to it.
- An evaluator reads the test case as
session["test_case"]. It holds the test case'sname,inputs, andexpected. For example, a code metric readssession["test_case"]["expected"]["answer"]to compare the pipeline's answer with the expected one. For how to write such a metric, see Write and Test an Evaluation Metric. - An Experiment never reuses an earlier evaluation for a replay of a row, because that evaluation might have read different expected data.
- If an Experiment compares a Candidate with the Recorded conversations Baseline on a Dataset that holds only rows, the Baseline is absent: nothing is replayed or scored for it, and the results show the Candidate alone, with no Baseline values or changes. On a Dataset that mixes rows and recorded conversations, the comparison works as usual.
- A Dataset holds at most 100 test cases.
- Optimization runs don't accept Datasets that hold rows yet.
To add a table to a Dataset, see Add Test Cases from a Table.
Metric Groups
A Metric Group is a named, ordered set of metrics in your workspace, for example "Answer quality" or "Cost and speed". Group metrics that belong together so you can pick the whole set by name instead of choosing metrics one by one each time.
Metric Groups work like this:
- A Metric Group holds metrics, not pinned versions. When you save a new version of a metric, every group that contains it uses the latest version.
- Results show metrics in the order you pick them.
- A group needs at least one metric and can hold up to 50. Scoring with an empty group would judge nothing, so the platform refuses it.
- A metric can belong to several groups.
- A new metric must join at least one Metric Group before you can save it, either an existing group or one you create together with the metric. This keeps every metric in your workspace reachable through a group.
- You can't delete a Metric Group while an optimization goal uses it.
You manage Metric Groups on the Metric Groups tab of the Metrics page. The AI assistant can also propose a Metric Group as part of a suggested experiment, and opens the same form pre-filled so you can review and adjust it before you save. For steps, see Manage Metric Groups.
Related Information
Was this page helpful?