How To Build Local LLM Apps With Ollama and SingleStore

In this article, we will explore how to use Ollama with LangChain and SingleStore using a Jupyter Notebook.

Akmal Chaudhri

CORE ·

May. 17, 24 · Tutorial

Likes (2)

Comment

Save

3.0K Views

In an era of heightened data privacy concerns, the development of local Large Language Model (LLM) applications provides an alternative to cloud-based solutions. Ollama offers one solution, enabling LLMs to be downloaded and used locally. In this article, we'll explore how to use Ollama with LangChain and SingleStore using a Jupyter Notebook.

The notebook file used in this article is available on GitHub.

Introduction

We'll use a Virtual Machine running Ubuntu 22.04.2 as our test environment. An alternative would be to use venv.

Create a SingleStoreDB Cloud Account

A previous article showed the steps required to create a free SingleStore Cloud account. We'll use Ollama Demo Group as our Workspace Group Name and ollama-demo as our Workspace Name. We’ll make a note of our password and host name. For this article, we'll temporarily allow access from anywhere by configuring the firewall under Ollama Demo Group > Firewall. For production environments, firewall rules should be added to provide increased security.

Create a Database

In our SingleStore Cloud account, let's use the SQL Editor to create a new database. Call this ollama_demo, as follows:

    SQL
   
   CREATE DATABASE IF NOT EXISTS ollama_demo;

Install Jupyter

From the command line, we’ll install the classic Jupyter Notebook, as follows:

    Shell
   
   pip install notebook

Install Ollama

We'll install Ollama, as follows:

    Shell
   
   curl -fsSL https://ollama.com/install.sh | sh

Environment Variable

Using the password and host information we saved earlier, we’ll create an environment variable to point to our SingleStore instance, as follows:

    Shell
   
   export SINGLESTOREDB_URL="admin:<password>@<host>:3306/ollama_demo"

Replace <password> and <host> with the values for your environment.

Launch Jupyter

We are now ready to work with Ollama and we’ll launch Jupyter:

    Shell
   
   jupyter notebook

Fill Out the Notebook

First, some packages:

    Shell
   
   !pip install langchain ollama --quiet --no-warn-script-location

Next, we’ll import some libraries:

    Python
   
 

   import ollama
from langchain_community.vectorstores import SingleStoreDB
from langchain_community.vectorstores.utils import DistanceStrategy
from langchain_core.documents import Document
from langchain_community.embeddings import OllamaEmbeddings
  

We'll create embeddings using all-minilm (45 MB at the time of writing):

    Python
   
   ollama.pull("all-minilm")

Example output:

    Plain Text
   
   {'status': 'success'}

For our LLM we'll use llama2 (3.8 GB at the time of writing):

    Python
   
   ollama.pull("llama2")

Example output:

    Plain Text
   
   {'status': 'success'}

Next, we’ll use the example text from the Ollama website:

    Python
   
 

   documents = [
    "Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels",
    "Llamas were first domesticated and used as pack animals 4,000 to 5,000 years ago in the Peruvian highlands",
    "Llamas can grow as much as 6 feet tall though the average llama between 5 feet 6 inches and 5 feet 9 inches tall",
    "Llamas weigh between 280 and 450 pounds and can carry 25 to 30 percent of their body weight",
    "Llamas are vegetarians and have very efficient digestive systems",
    "Llamas live to be about 20 years old, though some only live for 15 years and others live to be 30 years old"
]

embeddings = OllamaEmbeddings(
    model = "all-minilm",
)

dimensions = len(embeddings.embed_query(documents[0]))

docs = [Document(text) for text in documents]
  

We’ll specify all-minilm for the embeddings, determine the number of dimensions returned for the first document, and convert the documents to the format required by SingleStore.

Next, we’ll use LangChain:

    Python
   
 

   docsearch = SingleStoreDB.from_documents(
    docs,
    embeddings,
    table_name = "langchain_docs",
    distance_strategy = DistanceStrategy.EUCLIDEAN_DISTANCE,
    use_vector_index = True,
    vector_size = dimensions
)
  

In addition to the documents and embeddings, we’ll provide the name of the table we want to use for storage, the distance strategy, that we want to use a vector index, and the vector size using the dimensions we previously determined. These and other options are explained in further detail in the LangChain documentation.

Using the SQL Editor in SingleStore Cloud, let’s check the structure of the table created by LangChain:

    SQL
   
   USE ollama_demo;

DESCRIBE langchain_docs;

Example output:

    Plain Text
   
 

   +----------+------------------+------+------+---------+----------------+
| Field    | Type             | Null | Key  | Default | Extra          |
+----------+------------------+------+------+---------+----------------+
| id       | bigint(20)       | NO   | PRI  | NULL    | auto_increment |
| content  | longtext         | YES  |      | NULL    |                |
| vector   | vector(384, F32) | NO   | MUL  | NULL    |                |
| metadata | JSON             | YES  |      | NULL    |                |
+----------+------------------+------+------+---------+----------------+
  

We can see that a vector column with 384 dimensions was created for storing the embeddings.

Let’s also quickly check the stored data:

    SQL
   
   USE ollama_demo;

SELECT SUBSTRING(content, 1, 30) AS content, SUBSTRING(vector, 1, 30) AS vector FROM langchain_docs;

Example output:

    Plain Text
   
 

   +--------------------------------+--------------------------------+
| content                        | vector                         |
+--------------------------------+--------------------------------+
| Llamas weigh between 280 and 4 | [0.235754818,0.242168128,-0.26 |
| Llamas were first domesticated | [0.153105229,0.219774529,-0.20 |
| Llamas are vegetarians and hav | [0.285528302,0.10461951,-0.313 |
| Llamas are members of the came | [-0.0174482632,0.173883006,-0. |
| Llamas can grow as much as 6 f | [-0.0232818555,0.122274697,-0. |
| Llamas live to be about 20 yea | [0.0260244086,0.212311044,0.03 |
+--------------------------------+--------------------------------+
  

Finally, let’s check the vector index:

    SQL
   
   USE ollama_demo;

SHOW INDEX FROM langchain_docs;

Example output:

    Plain Text
   
 

   +----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
| Table          | Non_unique | Key_name   | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type       | Comment | Index_comment | Index_options                         |
+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
| langchain_docs |          0 | PRIMARY    |            1 | id          | NULL      |        NULL |     NULL |   NULL |      | COLUMNSTORE HASH |         |               |                                       |
| langchain_docs |          1 | vector     |            1 | vector      | NULL      |        NULL |     NULL |   NULL |      | VECTOR           |         |               | {"metric_type": "EUCLIDEAN_DISTANCE"} |
| langchain_docs |          1 | __SHARDKEY |            1 | id          | NULL      |        NULL |     NULL |   NULL |      | METADATA_ONLY    |         |               |                                       |
+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
  

We’ll now ask a question, as follows:

    Python
   
   prompt = "What animals are llamas related to?"
docs = docsearch.similarity_search(prompt)
data = docs[0].page_content
print(data)

Example output:

    Plain Text
   
   Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels

Next, we’ll use the LLM, as follows:

    Python
   
   output = ollama.generate(
    model = "llama2",
    prompt = f"Using this data: {data}. Respond to this prompt: {prompt}"
)

print(output["response"])

Example output:

    Plain Text
   
   Llamas are members of the camelid family, which means they are closely related to other animals such as:

1. Vicuñas: Vicuñas are small, wild relatives of llamas and alpacas. They are native to South America and are known for their soft, woolly coats.
2. Camels: Camels are also members of the camelid family and are known for their distinctive humps on their backs. There are two species of camel: the dromedary and the Bactrian.
3. Alpacas: Alpacas are domesticated animals that are closely related to llamas and vicuñas. They are native to South America and are known for their soft, luxurious fur.

So, in summary, llamas are related to vicuñas, camels, and alpacas.

Summary

In this article, we’ve seen that we can connect to SingleStore, store the documents and embeddings, ask questions about the data in the database, and use the power of LLMs locally through Ollama.

Data structure jupyter notebook SingleStore large language model artificial intelligence

Published at DZone with permission of Akmal Chaudhri. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

Trending

How To Build Local LLM Apps With Ollama and SingleStore

In this article, we will explore how to use Ollama with LangChain and SingleStore using a Jupyter Notebook.

Introduction

Create a SingleStoreDB Cloud Account

Create a Database

Install Jupyter

Install Ollama

Environment Variable

Launch Jupyter

Fill Out the Notebook

Summary

Related

Partner Resources