operators.spacy.SpacyTokenizer

class operators.spacy.SpacyTokenizer(text_field, tokens_field=None)

Preprocessor that parses document and adds tokens column.

Used by Sequence Tagging applications to add additional document metadata.

Parameters

Parameters

Name	Type	Default	Info
text_field	str		The field to parse with spacy.
tokens_field	Optional[str]	None	The field in which to store the tokens list object.