snorkelflow.sdk.BenchmarkExecution

class snorkelflow.sdk.BenchmarkExecution(benchmark_uid, benchmark_execution_uid, name, created_at, created_by)

Bases: object

\_\_init\_\_

__init__

__init__(benchmark_uid, benchmark_execution_uid, name, created_at, created_by)

Methods

__init__(benchmark_uid, ...)
export(filepath[, config])	Export information associated with this benchmark execution.

export

export

export(filepath, config=None)

Export information associated with this benchmark execution. The exported dataset includes:

• Per-datapoint evaluation information:

  • Evaluation scores, namely:

    • Parsed evaluator outputs

    • Rationale

    • Agreement with ground truth

  • Slice membership

• Benchmark metadata

• Execution metadata

• (CSV only) Uploaded user columns and ground truth

This export includes all datapoints without filtering or sampling. Some datapoints may have missing evaluation scores if the benchmark has not been executed against them (e.g. those in the test split).

Parameters

Parameters

Name	Type	Default	Info
filepath	str		The filepath to write the exported data to.
config	Union[JsonExportConfig, CsvExportConfig, None]	None	A JsonExportConfig or CsvExportConfig object. If not provided, JSON will be used by default. No additional configuration is required for JSON exports. For CSV exports, the following parameters are supported: sep: The separator between columns. Default is ,. quotechar: The character used to quote fields. Default is ". escapechar: The character used to escape special characters. Default is \.

Return type

Return type

None

Examples

>>> benchmark = Benchmark(123)
>>> execution = benchmark.list_executions()[0]
>>> execution.export("benchmark_execution.json")