snorkelflow.utils.open_file

snorkelflow.utils.open_file(path, mode='r', **kwargs)

Opens a file at the specified path and returns the corresponding file object.

Currently supports MinIO URLs in the following format: minio://{bucket}/{key}, local file paths, and S3 URLs.

Examples

>>> # Open a file in MinIO
>>> f = open_file("minio://my-bucket/my-key")

>>> # Open a file in MinIO outside of the in-platform Notebook
>>> f = open_file(
        "minio://my-bucket/my-key",
        key={MINIO-ACCESS-KEY},
        secret={MINIO-SECRET-KEY},
        client_kwargs={"endpoint_url": {MINIO-URL}}
    )

>>> # Open a local file
>>> f = open_file("./path/to/file")

>>> # Open a file in S3
>>> f = open_file("s3://my-bucket/my-key", key={YOUR-S3-ACCESS-KEY}, secret={YOUR-S3-ACCESS-KEY})

Parameters:

• path (str) – The path of the file to be opened

• mode (str, default: 'r') – 'w', 'rb', etc.

• kwargs (Optional[Dict[str, Any]]) – Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. These options are passed directly to fsspec.open.

Returns:

A file object opened in the specified mode

Return type:

OpenFile