Skip to main content
Version: 0.91

Using dedicated resources to speed up applications with large datasets

What will I learn?​

You will learn how enable dedicated resources in Snorkel to make your application faster. If you feel that your dataset is particularly large and is slowing down your app, consider using the dedicated resources feature to speed it up.

Dedicated resources caches your dataset in memory where it can be fetched and operated on quickly. To do this, Snorkel requires sufficient resources to cache your dataset in memory.

When would I need this?​

Consider using dedicated resources if your app is has the following dataset size and characteristics:

If your app feels sluggish or slow, consider using dedicated resources for a performance boost.

Who is eligible for this?​

This feature is currently available for Snorkel Hosted customers.

If you are an on-prem customer with a standard-sized deployment, you can still attempt to use dedicated resources by following the guide below.

  • The `studio-ray-worker` pod needs to have at least 8 CPU and 32 GB RAM, which is the default on a standard-sized deployment. If you do not have a standard-sized deployment, please contact your Snorkel representative to ask if you can use this feature.

If we cannot gather enough Dedicated Resources to enable the feature, Snorkel will display an error message that will tell you how many additional resources you need.

Please contact your Snorkel representative if you run into this message.

How do I use dedicated resources?​

Step 1: Click the Shared resources button at the top left of your Data Studio.

Step 2: Click Allocate dedicated resources.

Step 3: Continue using the app normally while you see the spinner that says Allocating resources....

Step 4: After at a minute, you'll see a green dot and Dedicated resources on the button.

Step 5: When you are done using the app, you can explicitly release resources by clicking Release dedicated resources.

After an hour of application inactivity, dedicated resources will release themselves back to the general pool.

Troubleshooting​

Because this is a Beta feature that is intended for Snorkel hosted customers, please reach out to your Snorkel representative if you run into challenges with using dedicated resources.

What if the text-box still says "Shared resources" after "Allocate dedicated Resources"?​

You may be encountering this error because you don't have enough resources on your deployment. If other people on your team are actively using dedicated resources, please ask them to click the Release dedicated resources button if they are no longer using them.

Check if you see an error message that tells you how much additional memory and CPU is needed to provision dedicated resources, and how much memory and CPU is remaining.

Provide this information to your Snorkel representative for faster triage