snorkelflow.operators.dask_extractor

class snorkelflow.operators.dask_extractor(*, input_schema, name=None, resources=None, resources_fn=None, output_schema=None)

Bases: dask_operator

Decorator to define Dask Extractor from a function.

Parameters:

• name (Optional[str], default: None) – Name of the Operator

• resources (Optional[Dict[str, Any]], default: None) – Resources passed in to f via kwargs

• resources_fn (Optional[Callable[[], Dict[str, Any]]], default: None) – A function for generating a dictionary of values passed to f via kwargs, that are too expensive to serialize as resources.

• input_schema (Dict[str, Any]) – Dictionary mapping from column to dtype, used to validate the dtypes of the input dataframe.

• output_schema (Optional[Dict[str, Any]], default: None) –

  Dictionary mapping from column to dtype, used to validate the dtypes of the output dataframe.

  If not None, then f must not delete any dataframe columns, and all new columns must be specified along with types in output_schema.

__init__(*, input_schema, name=None, resources=None, resources_fn=None, output_schema=None)

Methods

__init__(*, input_schema[, name, resources, ...])