Utilizing external models

Many FM Suite workflows rely upon or can make use of third-party external services to run inference over your data. To configure these third-party models, you first need to set up an account with the third-party. Once an account has been established, you can use the information on this page to configure the instance-wide settings for your particular service:

• Linking external services
• Configuring available models
• Supported models

Linking external services

We currently support connections to:

• HuggingFace
• OpenAI
• VertexAI language models
• Custom inference services that conform to the OpenAI API specification

Your API keys and secrets are securely managed by the SnorkelFlow secrets service. To list the services that are currently connected, use the following SDK command:

sf.list_secrets()

To add a connection to a new service, you can make use of the set secret command. Examples are shown below for each of the supported services:

• HuggingFace (API key can be found at https://huggingface.co/settings/tokens) {#hugging_face_sdk}

  sf.set_secret(key="huggingface::inference::api_token",
    value="**YOUR SECRET KEY**",
  )

• OpenAI (API key can be found at https://platform.openai.com/account/api-keys) {#openai_sdk}

  sf.set_secret("openai_api_key", "**YOUR API KEY**")

• VertexAI language models {#vertex_ai_sdk}

  sf.set_secret("vertexai_lm_location", "<<**your_project_location**>>")
  sf.set_secret("vertexai_lm_project_id", "<<**your_project_id**>>")
  sf.set_secret("vertexai_lm_credentials_json", "<<**your_json**>>")

• Custom inference service (conforms to the OpenAI API specification){#custom_inference_sdk}

  sf.set_secret("custom_inference_api_key", "**YOUR API KEY**")

Configuring available models

Once you have linked a service, you can add models that are served within those services. To list the models that are currently connected, use the following SDK command:

sf.get_external_model_endpoints()

To add models to SnorkelFlow, we need the following information for each model:

• Model Name: This can be set based on your preference. We recommend using the name of the model and provider (e.g., openai/gpt-4).
• Endpoint URL: An endpoint that we can hit to perform inference requests.
• Model Provider: The provider that is serving the model (HuggingFace, OpenAI, VertexAI LM, or custom inference service).
• SF Prompt Tools: A list of Snorkel Flow tools that the model should be made available for. This controls UX access to the model within the prompt builder workflows. For more information on prompting workflows, see Prompt builder.

The example below shows how to add the openai/gpt-4 model to Snorkel Flow:

from snorkelflow.client_v3.tdm.models import PromptTool, ExternalLLMProvider

sf.set_external_model_endpoint(
    "openai/gpt-4",  # Compatible chat completions model
    "https://api.openai.com/v1/chat/completions",  # Chat completions endpoint
    ExternalLLMProvider.OPENAI,  # Model provider
    [PromptTool.FREEFORM, PromptTool.EXTRACTIVE_LLM_QA],  # SF compatible tools
)

HuggingFace 🤗 inference endpoints

To add a HuggingFace model, you must first spin up your chosen HuggingFace model on their servers using https://ui.endpoints.huggingface.co/endpoints. After doing so, you will be given an endpoint URL for that model.

The example below shows how to set up a HuggingFace model with the extractive QA prompt tool:

sf.set_external_model_endpoint(
    "**MODEL NAME**",  # Set this to the model you have created the endpoint for
    "**ENDPOINT URL**",  # Set this to the URL for the model found in the Inference Endpoints Dashboard
    ExternalLLMProvider.HUGGINGFACE, # Model provider
    [PromptTool.EXTRACTIVE_QA], # SF compatible tools
)

OpenAI API

To add a Open AI model, you must have your API token set up and ensure that you are using the appropriate API endpoint. You'll want to specify either a chat completions endpoint for chat models or a completions endpoint for language models. Additional information about what API endpoint a model belongs to can be found at https://platform.openai.com/docs/guides/text-generation.

The example below shows how to set up a supported OpenAI model with the chat completions API endpoint (https://api.openai.com/v1/chat/completions):

sf.set_external_model_endpoint(
    "openai/gpt-3.5-turbo", # Compatible chat completions model
    "https://api.openai.com/v1/chat/completions", # Chat completions endpoint
    ExternalLLMProvider.OPENAI, # Model provider
    [PromptTool.FREEFORM, PromptTool.EXTRACTIVE_LLM_QA], # SF compatible tools
)

The example below shows how to set up a supported OpenAI model with the completions API endpoint (https://api.openai.com/v1/completions):

sf.set_external_model_endpoint(
    "openai/gpt-3.5-turbo-instruct", # Compatible completions model
    "https://api.openai.com/v1/completions", # Completions endpoint
    ExternalLLMProvider.OPENAI, # Model provider
    [PromptTool.FREEFORM], # SF compatible tools
)

VertexAI language models API

Once you have set up your API token, the example below shows how to set up a VertexAI model:

sf.set_external_model_endpoint(
    "vertexai_lm/text-bison@001",  # Model name
    "https://cloud.google.com/vertex-ai",  # Endpoint
    ExternalLLMProvider.VERTEXAI_LM,  # Model provider
    [PromptTool.FREEFORM],  # SF compatible tools
)

Custom Inference Service API

You can also add a model with custom inference service that conforms to the OpenAI API specification once you have set up your custom inference API token. Similar to the OpenAI setup, you will have to specify either a chat completions endpoint for chat models or a completions endpoint for language models. This inference service is also extensible to other foundation model providers that conform to OpenAI's API specification, such as Together AI.

The example below shows how to set up a supported model with the chat completions API endpoint:

sf.set_external_model_endpoint(
    "togetherai/Qwen/Qwen1.5-72B-Chat", # Chat model
    "https://api.together.xyz/v1/chat/completions", # Inference service chat endpoint
    ExternalLLMProvider.CUSTOM_INFERENCE_SERVICE, # Model provider
    [PromptTool.FREEFORM], # SF compatible tools
)

The example below shows how to set up a supported model with the completions API endpoint:

sf.set_external_model_endpoint(
    "togetherai/Qwen/Qwen1.5-72B", # Language model
    "https://api.together.xyz/v1/completions", # Inference service completions endpoint
    ExternalLLMProvider.CUSTOM_INFERENCE_SERVICE, # Model provider
    [PromptTool.FREEFORM], # SF compatible tools
)

Supported Models

We support prompting with specific model types from external providers for the different use cases in Snorkel Flow. The list provided below is not exhaustive. It only includes models that the Snorkel team has tested. Endpoints with compatible specifications to these providers could be swapped in as well. The table below provides a list of supported models for each use case:

Use Case	Prompt Tool	Supported Providers and Model types
Text classification	FREEFORM

• HuggingFace Text2Text and Text Generation models
• OpenAI models that are compatible with the chat completion API
• Vertex AI Palm and Gemini
• Custom inference service

| | Sequence tagging  | FREEFORM | (Beta) Reach out to your Snorkel representative.  | | EXTRACTIVE_QA | HuggingFace Question Answering models | | PDF extraction | EXTRACTIVE_DOCVQA | HuggingFace Document Question Answering models | | EXTRACTIVE_LLM_QA | OpenAI models that are compatible with the chat completion API |