Embeddings and Vector Databases: A Hands-On Guide!

At the heart of this revolution lies the concept of vector databases, a groundbreaking development that is reshaping how we handle complex data. Unlike traditional relational databases, these are uniquely equipped to manage and process high-dimensional vector data — which is intrinsic to many AI and machine learning applications. As we delve deeper into the era of advanced AI, vector databases are emerging as critical tools, offering unparalleled efficiency and accuracy in handling the vast, intricate datasets generated by generative AI models.

This article aims to explore the pivotal role of vector databases in the world of generative AI, highlighting their functionality, how they work, use cases, and a hands-on tutorial.

What Is a Vector Database?

A vector database is a kind of database that is designed to store, index, and retrieve data points with multiple dimensions, which are commonly referred to as vectors. Unlike databases that handle data (like numbers and strings) organized in tables, vector databases are specifically designed for managing data represented in multi-dimensional vector space. This makes them highly suitable for AI and machine learning applications, where data often takes the form of vectors like image embeddings, text embeddings, or other types of feature vectors.

These databases utilize indexing and search algorithms to conduct similarity searches, enabling them to rapidly identify the most similar vectors within a dataset. This capability is crucial for tasks like recommendation systems, image and voice recognition, and natural language processing since efficiently understanding and processing high-dimensional data plays a vital role. Consequently, vector databases represent an advancement in database technology tailored to meet the requirements of AI applications that rely heavily on vast amounts of data.

Vector Embeddings

When we talk about vector databases, we should definitely know what vector embeddings are — how data eventually gets stored in a vector database. Vector embeddings serve as numerical codes that encapsulate the key characteristics of objects, for example, songs in a music streaming app. By analyzing and extracting crucial features (like tempo and genre), each song is converted into a vector embedding through an embedding model.

This process ensures that songs with similar attributes have similar vector codes. A vector database stores these embeddings and, upon a query, compares these vectors to find and recommend songs with the closest matching features — facilitating an efficient and relevant search experience for the user.

How Does a Vector Database Work?

When a user query is initiated, various types of raw data, including images, documents, videos, and audio. All of this, which can be either unstructured or structured, is first processed through an embedding model. This model is often a complex neural network, translating data into high-dimensional numerical vectors and effectively encoding the data’s characteristics into vector embeddings — which are then stored into a a vector database like SingleStoreDB.

When retrieval is required, the vector database performs operations (like similarity searches) to find and retrieve the vectors most similar to the query, efficiently handling complex queries and delivering relevant results to the user. This entire process enables the rapid and accurate management of vast and varied data types in applications that require high-speed search and retrieval functions.

How Does a Vector Database Differ From a Traditional Database?

Let’s explore the difference between a vector database and a traditional database.

Vector databases represent a significant departure from traditional databases in their approach to data organization and retrieval. Traditional databases are structured to handle discrete, scalar data types like numbers and strings, organizing them in rows and columns.

This structure is ideal for transactional data but less efficient for the complex, high-dimensional data typically used in AI and machine learning. In contrast, vector databases are designed to store and manage vector data — arrays of numbers that represent points in a multi-dimensional space.

This makes them inherently suited for tasks involving similarity search, where the goal is to find the closest data points in a high-dimensional space, a common requirement in AI applications like image and voice recognition, recommendation systems, and natural language processing. By leveraging indexing and search algorithms optimized for high-dimensional vector spaces, vector databases offer a more efficient and effective way to handle the kind of data that is increasingly prevalent in the age of advanced AI and machine learning.

Vector Database Use Cases

Vector databases play a vital role in recommendation systems for businesses. For example, they can recommend items to a user depending on their browsing or buying behavior. They shine well even in fraud detection systems where they can detect anomalous patterns by comparing transaction embeddings against known profiles of fraudulent activity, thus enabling real-time fraud detection. Face recognition is an additional use case where vector databases store facial feature embeddings and help in security and surveillance.

They can even help organizations with customer support by responding to similar queries with pre-determined or varied responses. Market research is another area where vector databases do well by analyzing customer feedback and social media posts, converting them into text embeddings to do sentiment analysis and trend spotting — gaining even more business insights.

SingleStoreDB as a Vector Database

The robust vector database capabilities of SingleStoreDB are tailored to seamlessly serve AI-driven applications, chatbots, image recognition systems and more. With SingleStoreDB, the necessity for maintaining a dedicated vector database for your vector-intensive workloads becomes obsolete.

Diverging from conventional vector database approaches, SingleStoreDB takes a novel approach by housing vector data within relational tables alongside diverse data types. This innovative amalgamation empowers you to effortlessly access comprehensive metadata and additional attributes pertaining to your vector data, all while leveraging the extensive querying prowess of SQL.

SingleStoreDB has been meticulously architected with a scalable framework, ensuring unfaltering support for your burgeoning data requirements. Say goodbye to limitations and embrace a solution that grows in tandem with your data demands.

Vector Database Tutorial Using SingleStoreDB

Prerequisites

• Sign up for your free SingleStoreDB Cloud trial. We will use SingleStore as our vector database.
• Postman account to create embeddings
• OpenAI account to get our OpenAI API key

Once you sign in to your OpenAI account, go to the API tab shown here.

Next, go to the embeddings tab.

We are going to start off with API requests for embeddings. To do that, we will need to go to the API References page. Move to the ‘Embeddings’ tab under the API References page to see how to create embeddings.

Now, let’s create API requests for our embeddings. For this, we need a tool like Postman. You can sign up and create a workspace in your Postman account.

Now, get our API URL `https://api.openai.com/v1/embeddings` and paste it in the Postman URL section, authorizing it using your OpenAI API Key.

It is time to create our first embedding. We just need a model and the input parameters as shown in the OpenAI document page.

For the model, we will be using “text-embedding-ada-002” and any text for the input.

Let’s do this. Head over to the Postman dashboard and create a ‘body’ request. Under the body, select ‘raw’ and then select ‘JSON’ to pass a JSON object.

Mention the model and input.

Send the post request to OpenAI. You should see a similar response as shown here.

The numbers you see are the vectors for the embedding. To store these vector embeddings, you need a robust database — that is where SingleStoreDB shines. Let’s create a free SingleStoreDB Cloud account.

Next, create a workspace, as shown here.

You can see your created workspace and sample/default database attached.

Let’s create a database.

The newly created database can be seen in the dashboard.

You can go inside this database to see the contents. In this case, since we haven’t fed any data yet, you won’t see anything.

Let’s create some tables in our database. Head over to the ‘SQL Editor’ as shown. You can run some simple SQL commands to create tables.

Make sure to select the database under the ‘Select Database’ tab. Select the database that we created a while back.

Write a simple SQL query to create a table and data type.

Now, let’s insert vector data into this database. This is where we are going to feed the embedding data we received from Postman. Head back to the SQL Editor and write the following SQL Query.

We will be using values with reference to our ‘Hello World’ input from Postman. Insert the huge block of numbers we received from Postman inside the JSON_ARRAY_PACK.

Copy these numbers from Postman. Paste the huge block of numbers into your values and run the command.

You can see the results being fed into our database.

That’s it! This is how you can create different embeddings using Postman from different inputs and add the generated vector embeddings to your newly created database.

Use the same approach as earlier to store them under our database. Add your own inputs and content to convert them into vectors. As you can see here, we have added different inputs and stored the generated vector embeddings into our database.

Let’s do some experimentation now by searching some specific terms and retrieving relevant data from our database. Let’s go back to the Postman dashboard and create an embedding for the term ‘OpenAI.’

Next, we will perform a search in the database against the existing embeddings. This should give us results, with the list of closest similarities being at the top. Let’s head over to the SQL editor and run this query as shown.

Paste the embedding (huge block of numbers) inside JSON_ARRAY_PACK and run the command.

In the preceding image, you can see the similarity scores and rankings for the keyword ‘OpenAI’. Similarly, you can see similarity scores for any keywords. This way, SingleStoreDB can be used as an efficient vector database for your business.