Skip to main content
Version: 25.8

snorkelai.sdk.develop.ErrorAnalysis

class snorkelai.sdk.develop.ErrorAnalysis(provenance, error_analysis_run_uid)

Bases: object

This class provides methods for creating, monitoring, and retrieving results from error analysis clustering runs. The clustering algorithm identifies patterns in LLM evaluation failures and groups similar errors together.

Parameters​

NameTypeDefaultInfo
provenanceDict[str, int]Tracking information containing the benchmark_uid and criteria_uid that uniquely identifies the inputs used to create this cluster analysis.
error_analysis_run_uidintUnique identifier for this specific error analysis run.

__init__​

__init__(provenance, error_analysis_run_uid)

Initialize an ErrorAnalysis instance.

Parameters​

NameTypeDefaultInfo
provenanceDict[str, int]Tracking information that uniquely identifies the inputs used to create this cluster analysis.
error_analysis_run_uidintUnique identifier for this specific cluster analysis run.

Methods

__init__(provenance, error_analysis_run_uid)Initialize an ErrorAnalysis instance.
create(prompt_execution_uid, *[, sync])Create and trigger error analysis clustering job for LLM evaluation results.
delete()Delete this error analysis run and all associated cluster data.
get(error_analysis_run_uid)Retrieve an existing error analysis by its unique run identifier.
get_cluster_membership(cluster_uid)Fetch datapoint membership for a specific cluster.
get_clusters()Fetch clusters from a completed error analysis.
get_latest(prompt_execution_uid)Get the most recent error analysis run for a specific prompt execution.

create​

classmethod create(prompt_execution_uid, *, sync=False)

Create and trigger error analysis clustering job for LLM evaluation results.

Parameters​

NameTypeDefaultInfo
prompt_execution_uidintUnique identifier of the prompt execution.
syncboolFalseIf True, method blocks until clustering completes and returns ready ErrorAnalysis. If False, returns immediately with ErrorAnalysis that requires waiting to get clusters.

Returns​

An ErrorAnalysis instance representing the clustering job. If sync = False, get_clusters will fail until analysis is complete. If sync = True, results are immediately available.

Return type​

ErrorAnalysis

Raises​

  • ValueError – If benchmark, prompt execution or criteria don’t exist or are not valid

  • ValueError – If prompt execution has no evaluation results or insufficient error cases to cluster

Examples​

Create error analysis asynchronously:

>>> error_analysis = ErrorAnalysis.create(
...     prompt_execution_uid=456,
...     sync=False
... )

Create error analysis synchronously:

>>> error_analysis = ErrorAnalysis.create(
...     prompt_execution_uid=456,
...     sync=True
... )

delete​

delete()

Delete this error analysis run and all associated cluster data.

Raises​

ValueError – If the error analysis run has already been deleted or does not exist

Return type​

None

Example​

>>> error_analysis = ErrorAnalysis.get(error_analysis_run_uid=42)
>>> error_analysis.delete()

get​

classmethod get(error_analysis_run_uid)

Retrieve an existing error analysis by its unique run identifier.

Parameters​

NameTypeDefaultInfo
error_analysis_run_uidintThe unique identifier of the error analysis run to retrieve.

Returns​

The ErrorAnalysis instance for the specified run, ready for status checks and results retrieval

Return type​

ErrorAnalysis

Raises​

ValueError – If no error analysis run exists with the given ID

Example​

>>> error_analysis = ErrorAnalysis.get(error_analysis_run_uid=42)

get_cluster_membership​

get_cluster_membership(cluster_uid)

Fetch datapoint membership for a specific cluster.

Parameters​

NameTypeDefaultInfo
cluster_uidintUnique identifier of the cluster to get membership for.

Returns​

DataFrame containing all the datapoints in the cluster with columns: - datapoint_uid: int - … (additional columns as available)

Return type​

pd.DataFrame

Raises​

ValueError – If cluster uid does not exist

Example​

>>> clusters = error_analysis.get_clusters()
>>> cluster_uid = clusters[0]['cluster_uid']
>>> membership_df = error_analysis.get_cluster_membership(cluster_uid)

get_clusters​

get_clusters()

Fetch clusters from a completed error analysis.

Returns​

List of cluster dictionaries, each containing: - cluster_uid: int - name: str - description: str - improvement_strategy: str - examples: List[str] - datapoint_count: int

Return type​

List[Dict[str, Any]]

Raises​

  • RuntimeError – If called before analysis is complete

  • ValueError – If analysis failed or was deleted

Examples​

Get clusters from error analysis:

>>> clusters = analysis.get_clusters()

get_latest​

classmethod get_latest(prompt_execution_uid)

Get the most recent error analysis run for a specific prompt execution.

Parameters​

NameTypeDefaultInfo
prompt_execution_uidintThe unique identifier of the prompt execution.

Returns​

The most recent ErrorAnalysis for the prompt execution or None if no error analysis has been run for this prompt execution

Return type​

ErrorAnalysis or None

Raises​

ValueError – If prompt execution does not exist

Example​

>>> latest_analysis = ErrorAnalysis.get_latest(
...     prompt_execution_uid=456
... )