Version: 0.93

snorkelflow.client.ctx.SnorkelFlowContext

class snorkelflow.client.ctx.SnorkelFlowContext(tdm_client, minio_args=None, studio_client=None, workspace_name='default', set_global=True, debug=False)

Bases: object

The SnorkelFlowContext object provides client context for the Snorkel Flow SDK. It allows the Snorkel Flow SDK to recognize a Snorkel Flow instance by identifying Snorkel Flow’s essential API services (TDM, Studio, MinIO) via user-provided parameters, a YAML config file, or a Snorkel Flow API key.

Examples

import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_kwargs(...)

The keyword arguments are presented in the documentation below. Alternatively, if you have a .snorkel-flow.yaml config file locally, you can use it to create a context:

import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_user_config("path/to/.snorkel-flow.yaml")

__init__(tdm_client, minio_args=None, studio_client=None, workspace_name='default', set_global=True, debug=False): Initialize a SnorkelFlowContext.

Methods

`__init__`(tdm_client[, minio_args, ...])	Initialize a SnorkelFlowContext.
`from_kwargs`([tdm_scheme, tdm_host, ...])	Initialize a SnorkelFlowContext from keyword arguments.
`from_user_config`([user_config_yaml, ...])	Method to create a SnorkelFlowContest object from a `.snorkel-flow.yaml` file.
`get_global`()	Retrieve the global SnorkelFlowContext object.
`get_minio_client`()	Retrieve an S3 resource for the Minio client.
`set_debug`(debug)	Set the verbosity of warnings, details, and stacktraces
`set_global`([ctx])	Set a SnorkelFlowContext object globally.

Attributes

workspace_name

SnorkelFlowContext objects are only scoped to work in a particular workspace.

classmethod from_kwargs(tdm_scheme='http', tdm_host='tdm-api', tdm_port=8686, tdm_url_prefix='', minio_scheme='http', minio_host='minio', minio_port=9001, minio_access_key=None, minio_secret_key=None, studio_scheme='http', studio_host='studio-api', studio_port=8484, studio_url_prefix='', workspace_name='default', api_key=None, set_global=True, debug=False)

Initialize a SnorkelFlowContext from keyword arguments.

Examples

# Instantiate with default kwargs
import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_kwargs()

# Instantiate with custom kwargs
import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_kwargs(
    tdm_port=8080,
    minio_port=9000,
    minio_access_key="minio",
    minio_secret_key="minio123",
    studio_port=8080,
    workspace_name="my-workspace",
)

# Instantiate with an API key
import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_kwargs(
    api_key="my-api-key",
)

Parameters:

tdm_scheme (str, default: 'http') – The scheme to use for the TDM API.
tdm_host (str, default: 'tdm-api') – The host to use for the TDM API.
tdm_port (int, default: 8686) – The port to use for the TDM API. Contact an administrator for information.
tdm_url_prefix (str, default: '') – The url prefix to use for the TDM API.
minio_scheme (str, default: 'http') – The scheme to use for MinIO.
minio_host (str, default: 'minio') – The host to use for MinIO.
minio_port (int, default: 9001) – The port to use for MinIO. Contact an administrator for information.
minio_access_key (Optional[str], default: None) – The access key to use for MinIO. If not provided, looks for the access key as an environment variable.
minio_secret_key (Optional[str], default: None) – The secret key to use for MinIO. If not provided, looks for the access key as an environment variable.
studio_scheme (str, default: 'http') – The scheme to use for the Studio API.
studio_host (str, default: 'studio-api') – The host to use for the Studio API.
studio_port (int, default: 8484) – The port to use for the Studio API. Contact an administrator for information.
studio_url_prefix (str, default: '') – The url prefix to use for the Studio API.
workspace_name (str, default: 'default') – The workspace name, which determines the workspace a dataset and application will be created in and queried from.
api_key (Optional[str], default: None) – The API key to use for the TDM and Studio APIs.
set_global (bool, default: True) – Whether to set the context as the global context, accessible across all notebooks in-platform.
debug (bool, default: False) – Whether to enable debug mode. Debug mode will print more verbose errors to the screen inside of the Python notebook.

Returns:

A SnorkelFlowContext object that can be used to interact with the MinIO, TDM, and Studio clients directly.

Return type:

SnorkelFlowContext

classmethod from_user_config(user_config_yaml=None, protocol='http', set_global=True, debug=False)

Method to create a SnorkelFlowContest object from a .snorkel-flow.yaml file.

Examples

import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_user_config("path/to/your/.snorkel-flow.yaml")

Parameters:

user_config_yaml (Optional[str], default: None) – Path to the .snorkel-flow.yaml file.
protocol (str, default: 'http') – The scheme to use for TDM, Studio, and MinIO.
set_global (bool, default: True) – Whether to use this YAML file to determine the context globally
debug (bool, default: False) – Whether to enable debug mode. Debug mode will print more verbose errors to the screen inside of the Python notebook.

Returns:

A SnorkelFlowContext object that can be used to interact with the MinIO, TDM, and Studio clients directly.

Return type:

SnorkelFlowContext

classmethod get_global()

Retrieve the global SnorkelFlowContext object.

Examples

import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.get_global()

Returns:: A SnorkelFlowContext object that can be used to interact with the MinIO, TDM, and Studio clients directly.
Return type:: SnorkelFlowContext
Raises:: AttributeError – If no global context has been set.

get_minio_client()

Retrieve an S3 resource for the Minio client. All methods for the retrieved resource can be found in the Boto3 documentation.

Examples

import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_kwargs()
mc = ctx.get_minio_client()
mc.create_bucket(Bucket="my-bucket")

# Create a new bucket
mc = ctx.get_minio_client()
mc.create_bucket(Bucket="my-bucket")

# List all objects in a bucket
bucket = mc.Bucket("my-bucket")
for obj in bucket.objects.all():
    print(obj)

Return type:: S3.ServiceResource
Raises:: RuntimeError – If no valid connection to the MinIO client can be established.

set_debug(debug)

Set the verbosity of warnings, details, and stacktraces

Parameters:: debug (bool) – If True, SDK operations will print more verbose errors to the screen inside of the Python notebook.
Return type:: None

classmethod set_global(ctx=None)

Set a SnorkelFlowContext object globally. This context object will be used by all Snorkel Flow SDK functions.

Examples

import snorkelflow.client as sf
ctx = sf.SnorkelFlowContext.from_kwargs(...)
sf.SnorkelFlowContext.set_global(ctx)

Parameters:: ctx (Optional[SnorkelFlowContext], default: None) – A context to set as global. If no context is provided, will attempt to create one with default parameters.
Return type:: None

property workspace_name: str

SnorkelFlowContext objects are only scoped to work in a particular workspace.

Returns:: The name of the workspace belonging to this context object.
Return type:: str