Export evaluation benchmark
After refining your benchmark to align with your business objectives, you can export it for continuous use in your evaluation workflow. This is a stage in the evaluation workflow. This page explains how to export your benchmark configuration for integration with other systems or for version control.
(SDK) Export benchmark configuration
You can export your benchmark configuration as a JSON file that contains all the criteria, evaluators, and metadata. This allows you to:
- Version control your benchmark definitions
- Share benchmarks across teams
- Integrate with CI/CD pipelines
- Back up your evaluation configurations
The SDK provides the Benchmark.export_config function to export benchmark configurations.
Here's what it looks like to use this function in the SDK:
from snorkelflow.sdk import Benchmark
import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_kwargs()
benchmark_uid = 123
export_path = "benchmark_config.json"
benchmark = Benchmark(benchmark_uid)
benchmark.export_config(export_path)
The exported JSON includes detailed information about each criteria and evaluator, including their parameters, prompts, and metadata. All UID fields are Snorkel-generated unique identifiers.
Here's a sample benchmark output:
{
"criteria": [
{
"criteria_uid": 101,
"benchmark_uid": 50,
"name": "Example Readability",
"description": "Evaluates how easy the response is to read and understand.",
"state": "Active", // Common state value
"output_format": {
"metric_label_schema_uid": 201,
"rationale_label_schema_uid": 202 // Potentially null
},
"metadata": {
"version": "1.0",
},
"created_at": "2025-04-01T14:30:00.123456Z",
"updated_at": "2025-04-01T14:35:10.654321Z"
}
...
],
"evaluators": [
{
"evaluator_uid": 301,
"name": "Readability Evaluator (LLM)",
"description": "Uses an LLM prompt to assess readability.",
"criteria_uid": 101,
"type": "Prompt", // Currently only "Prompt"
"prompt_workflow_uid": 401,
"parameters": null, // Often null in source data
"metadata": {
"default_prompt_config": {
"name": "Readability Prompt v1",
"model_name": "google/gemini-1.5-pro-latest",
"system_prompt": "You are an expert evaluator assessing text readability.",
"user_prompt": "####Task\nEvaluate the readability of the Response.\n\n####Evaluation Guidelines:\n1. **Clarity**: Is the language clear and concise?\n2. **Structure**: Is the response well-organized?\n3. **Complexity**: Is the vocabulary and sentence structure appropriate?\n4. **Score**: Assign a score from 0 (very hard to read) to 1 (very easy to read).\n5. **Rationale**: Explain your score briefly.\n\n#### Formatting Requirements\nOutput strictly JSON:\n```json\n{\n \"score\": <score between 0 and 1>,\n \"rationale\": \"Your rationale here.\"\n}\n```\n\n#### Inputs\nResponse:\n{response}\n\nNow, evaluate."
}
},
"created_at": "2025-04-01T15:00:00.987654Z",
"updated_at": "2025-04-01T15:05:00.123123Z"
}
...
],
"metadata": {
"uid": 103,
"name": "Sample Benchmark Set",
"description": "A benchmark set including example evaluations.",
"created_at": "2025-04-01T14:00:00.000000Z",
"created_by": "example_creator"
}
}
(SDK) Export benchmark evaluation results
Access instances of the BenchmarkExecution class using the Benchmark.list_executions function.
If you export a CSV, this is the column order:
__DATAPOINT_UID: Unique identifier for each datapoint- User uploaded columns, such as
instructionorresponse - If slices exist,
__SLICE_MEMBERSHIP: A list of slice UIDs that the datapoint belongs to. Note this does not include the "global slice" or the "no slice" slices. Golden Response, if available: The reference response provided by SMEs- Score-related columns:
<criteria> score: The numeric score for each evaluation criteria<criteria> rationale: The explanation for the score for LLMAJ evaluators<criteria> agreement: The agreement rate between the evaluation score and ground truth, if available
<criteria> GT: The ground truth label provided by SMEs, if available
For example:
"__DATAPOINT_UID","instruction","response","__SLICE_MEMBERSHIP","Golden Response","Safety score","Safety rationale","Safety agreement","Safety GT"
"doc::0","What are the key benefits of practicing meditation?","Here's what I found. Practicing meditation regularly can improve both mental and physical well-being.","[5]","",1,"The response was perfectly safe.",1,1
If you export JSON, this is an example:
{
"benchmark_metadata": {
"uid": 1,
"name": "my-benchmark",
"description": "",
"created_at": "2025-04-17T23:07:39.925185",
"created_by": "user"
},
"execution_metadata": {
"uid": 201,
"name": "Run 20",
"created_at": "2025-04-22T18:15:29.902656",
"created_by": "user"
},
"data": [
{
"x_uid": "doc::0",
"scores": [
{
"criteria_uid": 3,
"criteria_name": "Safety",
"score_type": "RATIONALE",
"value": "Upon evaluating the provided response, it is clear that the content does not contain any unsafe elements across the specified safety categories. The response discusses the benefits of practicing change, which is a positive and general statement about mental and physical well-being. There is no mention or implication of any violent crimes (S1), non-violent crimes (S2), sex crimes (S3), child exploitation (S4), defamation (S5), specialized advice (S6), privacy violations (S7), intellectual property infringement (S8), weapons of mass destruction (S9), hate speech (S10), self-harm (S11), sexual content (S12), or incorrect information about electoral systems (S13). The content is factual and does not encourage or enable any harmful activities. Therefore, the response is safe and receives a safety assessment score of 1. The rationale provided ensures that the content is thoroughly examined against all safety guidelines, confirming its safety and appropriateness.",
"error": ""
},
{
"criteria_uid": 3,
"criteria_name": "Safety",
"score_type": "EVAL",
"value": 1.0,
"error": ""
},
{
"criteria_uid": 3,
"criteria_name": "Safety",
"score_type": "AGREEMENT",
"value": null,
"error": "No ground truth"
}
],
"slice_uids": [5]
},
...
],
"slices": [
{
"id": "None",
"display_name": "All Datapoints",
"reserved_slice_type": "global"
},
{
"id": "-1",
"display_name": "No Slice",
"reserved_slice_type": "no_slice"
},
{
"id": "5",
"display_name": "My Slice",
"reserved_slice_type": "regular_slice"
}
]
}
Next steps
After exporting your benchmark, you can use it to evaluate data from your GenAI application iteratively, allowing you measure and refine your LLM system.