(Beta) Prompt development Overview

note

This is a beta feature available to customers using a Snorkel-hosted instance of Snorkel Flow. Beta features may have known gaps or bugs, but are functional workflows and eligible for Snorkel Support. To access beta features, contact Snorkel Support to enable the feature flag for your Snorkel-hosted instance.

Prompt development, sometimes called prompt engineering, is the process of designing and refining inputs to guide AI models, such as large language models (LLMs), to produce high-quality, task-specific outputs. In Snorkel Flow, prompt development workflows empower users to create, experiment with, and optimize prompts iteratively. This ensures efficient, precise, and reproducible outcomes for downstream tasks.

A prompt development workflow begins with uploading a text dataset, selecting an LLM, and crafting system and/or user prompts to tailor the model's response. With prompt versioning, users can iterate on their designs, enabling continuous improvement of AI-driven results.

Prompt development enables users to harness the full potential of AI models by guiding them to produce accurate, efficient, and contextually relevant outputs. By optimizing prompts, it reduces the need for extensive fine-tuning, accelerates workflows, and ensures models align with specific business goals or user needs.

Components of Prompt Development

Input Dataset

A prompt development workflow uses an input dataset. This dataset serves as context or examples for the LLM to work with. An input dataset for prompt development should contain relevant data points or scenarios that the LLM can use to align its output with your goals. These are types of input datasets:

• FAQ generation: Provide question-answer pairs.
• Classification tasks: Include customer conversations or labeled examples.

Best practices

When creating input datasets, it is important to follow best practices:

• Use clear column names that map to LLM processes and predictions. For example, User Input, Context, and Expected Response.
• Make sure you select the train split when uploading the input dataset.

note

Before creating a prompt development workflow, upload the dataset you plan to use. Learn more about uploading datasets.

LLMs

The prompt development workflow allows you to experiment with different LLMs to identify the model and prompt combination that delivers the highest quality results.

Prompts

There are multiple types of prompts in prompt development:

• System prompt: Sets the overarching tone, rules, or constraints for the LLM. Examples include:

  • Instructing the model to adopt a persona. You are a financial assistant.
  • Ensuring responses adhere to specific styles or formats. Format this using the Google Developer Style Guide.

• User prompt: Focuses on the primary task or request. These prompts add task-specific instructions, often referencing input dataset columns dynamically. Examples include:

  • For simple tasks not requiring global rules or a consistent persona. Translate the following text into French: {Text}.
  • When the task or output is fully defined in the user prompt. Extract all ingredients from the following recipe: {Recipe}.

Using multiple prompts together

While user prompts can suffice for simple tasks, combining system and user prompts is recommended for:

• Consistency: The system prompt ensures predictable behavior. For example:

  • System Prompt: You are a data assistant summarizing technical documents.
  • User Prompt: Summarize the following text: {Text}.

• Complex workflows: System prompts provide global instructions while user prompts handle granular tasks. For example:

  • System Prompt: You are an expert in natural language processing. Always respond in JSON format.
  • User Prompt: Extract named entities from the following text: {Text}.

Features and benefits

Key features

• Workflow Management: Create, view, and manage prompt workflows via the Prompts page.
• Dataset Selection: Choose datasets with clearly displayed columns.
• Prompt Experimentation: Test different LLMs, system prompts, and user prompts.
• Prompt Versioning: Save, view, and compare prompt iterations.

Workflow capabilities

1. Run Prompts: Execute prompts and view LLM responses.
2. Iterative Refinement: Update prompts iteratively for continuous improvement.
3. Comparison View: Analyze multiple LLM responses side-by-side.
4. Output Integration: View output datasets on the Datasets page.

Known limitations

These are the known limitations for prompt development:

• Only text datasets are supported.
• Datasets can contain a maximum of 400 rows.
• We assume 100% of your data belongs to train split.
• Data is viewable for only one data source at a time.

What's next?

You can learn to create a prompt development workflow.