Article

Airbyte x Radiant: How to double your token limits without any new code

•

January 4, 2024

•

4 min read

With Airbyte you can enable semantic search over many data sources by using embedding models and a vector database. Airbyte comes out of the box with support for vector databases (like Qdrant, Pinecone, Weaviate, Chroma) and embedding models such as OpenAI, Azure OpenAI and Cohere.

One common problem we have seen from people running embeddings at scale is how to avoid hitting rate limits.

For example, when ingesting social media data, the amount of data can fluctuate quite significantly or multiple teams sharing the same embedding model can lead to resource congestion.

Radiant is a platform that enables load balancing between different embedding models. With Radiant, multiple Azure tenants can be used as one. Additionally, syncs do not stop if one embedding model suffers an outage.

In this article we show you how to set it up with Airbyte. We will set up a simple sync from an S3 bucket to a Qdrant vector database using Radiant to load balance between embedding models.

Setup

Set up a Source

First, we create a new Source in Airbyte. For this example, we use a bucket which contains Arxiv papers (about AI research of course). We want to index those papers into QDrant to make it easier to find sources for specific questions or further on in our RAG applications. We use the Airbyte “Document File Type Format” steam to automatically pick up our PDF files.

Radiant

Before we can set up our destination in Airbyte, we configure Radiant to receive embedding requests. You can request access to the Radiant platform on the Radiant website. Once you have access, set up a provider that serves the embedding model.

For example, to add OpenAI as a provider, click the “Add Provider”, select OpenAI from the dropdown menu and enter your OpenAI API key. In our example we are setting up both an OpenAI and Azure OpenAI embedding model.

Warning:

You need to make sure to use instances of the same model in your embedding step. Different embedding models that do not share the same embedding space are NOT compatible with each other.

Once we have set up multiple embedding model providers, we create an application that uses them. In Radiant, applications are an easy way to distinguish between different use cases for LLMs and embedding models. Each application comes with its own URL which makes it easy to integrate with existing applications or third party software. We create a new application for our paper indexing use case and configure it to use multiple embedding models:

Now that Radiant is ready and configured, copy the base URL and keep it handy. You can find it for example in the code examples page. In our case it is

openai.api_base = "https://[radiant instance].deployradiant.com/arxiv-paper-indexing/openai"

Destination

For the purpose of this example we are going to use the Qdrant destination. If you want to test it yourself, setting up a local Qdrant is super easy and straightforward through docker. For the Chunk Size we use 256. This is the number of tokens embedded at once.

For the embedding configuration we point the Destination at our Radiant URL via the “OpenAI-compatible” configuration. The embedding dimension needs to match the embedding model we are using. In our case for Ada it is 1024.

Lastly, we need to provide the location of our Qdrant database and the collection we want to store it under.

When we hit Test and Save we see a test request being sent to Radiant and can confirm everything is working.

Success

Now that we have our connection set up we can go ahead and run it. After a few minutes everything should be successful and your Vector DB is updated.

In Radiant we can see how many tokens we have been used during the indexing and can see that we have used both providers when we check the logs.

Conclusion

With this set up in place, we can now run our embeddings connections in Airbyte while using multiple embedding models. This allows us to get around rate limitations easily as well as a single model provider having an outage.

Instead of having a single point of failure we now have resilience in our connection. Please check out Airbyte for more options when it comes to data sources and vector databases and Radiant when it comes to embedding models and metrics around them.

Next steps

How to setup more complex pipelines with more transformation steps between vector DB and data source check out https://airbyte.com/blog/eltp-extending-elt-for-modern-ai-and-analytics
Check out different open source embedding models from huggingface https://huggingface.co/spaces/mteb/leaderboard

https://airbyte.com/blog/airbyte-now-supports-vector-databases-powered-by-langchain

Limitless data movement with free Alpha and Beta connectors

Introducing: our Free Connector Program ->

The data movement infrastructure for the modern data teams.

Try a 14-day free trial