Explore Open WebUI: Your Offline AI Interface

Are you looking for a user-friendly, self-hosted AI interface designed to operate entirely offline? Look no further; Open WebUI might be what you are looking for. In this blog, you will dive into some nice features of Open WebUI. Enjoy!

Introduction

Quite some inference engines exist which can run locally, for example, Ollama, LMStudio, LocalAI, Jan, and many others. Some of them also come with a Graphical User Interface (GUI). However, these run locally on the same machine as the inference engine. What if you would like to offer a ChatGPT-like user interface within your company that runs entirely offline? The inference engine runs on dedicated machines with GPUs (on-premise or in a private cloud), and the GUI is a web application available to your users. The advantage of this setup is that all your data does not leave your company and is not shared with any cloud vendor. The privacy of your data is secured with this setup. 

The disadvantage is that you might not be able to run large models and that you need to make a trade-off between quality and performance. However, Exo and JLama are initiatives that are working on AI clustering, which will make it possible to run large models locally. It is just a matter of time before this functionality will be available.

In the meanwhile, Open WebUI provides you with a web application that offers quite some functionality to run your AI solution entirely offline. Let’s explore some of its features in this blog. Check out the official documentation of Open WebUI, which is not complete but offers good information about its features.

Sources used in this blog are available on GitHub.

Prerequisites

Prerequisites for reading this blog are:

• Basic knowledge of Docker and Docker Compose;
• Basic knowledge of large language models (LLMs);
• Basic knowledge of retrieval-augmented generation (RAG). More info can be found in a previous blog.

3. Installation

Several methods are available for installing Open WebUI. In this section, the setup with a locally running Ollama instance is described together with Open WebUI running in a Docker container. For ease of use, the Docker command is converted to a compose.yaml file.

version: '3'
 
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    volumes:
      - open-webui:/app/backend/data
 
volumes:
  open-webui:

Start the container from where the compose file is located.

$ docker compose up

After startup, Open WebUI is accessible via http://localhost:3000/.

Create an admin account.

After a successful login, the webpage shows a blank screen. After approximately 5 minutes, an error occurs in the logs.

open-webui  | ERROR [open_webui.routers.ollama] Connection error: Cannot connect to host host.docker.internal:11434 ssl:default [Connect call failed ('172.17.0.1', 11434)]

After this error, the release notes were shown.

However, it was not possible to select a model, although models are available in the locally running Ollama instance.

Shut down the container with CTRL+C or by entering the command:

$ docker compose down

Apparently, Open WebUI is not able to access the running Ollama instance on the host. Therefore, you can bind the container to your host network, as described here.

The compose file becomes the following.

version: '3'
 
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    network_mode: host
    volumes:
      - open-webui:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://localhost:11434
 
volumes:
  open-webui:

Start the container again.

$ docker compose up

Note that Open WebUI is now accessible via http://localhost:8080/.

My First Prompt

Ollama already contains some models. If you have not installed any, you can install llama3.2:3b which is a small and good model to start with.

$ ollama pull llama3.2:3b

Now that the connection with Ollama is working, you will see all available models in Open WebUI.

As a first prompt, you ask the model to tell a joke.

And the model returns a joke.

This works quite similarly to any other AI service, but the difference is here that everything is running locally. No data is sent outside of your laptop!

Knowledge

A very interesting feature is the knowledge feature. This allows you to add your own documents as context to the model. It is, in fact, retrieval-augmented generation (RAG). You can add your documents by clicking the workspace in the left menu. This will show you some tabs under the Knowledge tab.

In a previous blog, RAG was implemented and evaluated by using three markdown documents containing data in table format about songs and albums of Bruce Springsteen. Let’s add these documents.

Click the +-sign at the right to create a new knowledge base.

Drag and drop each file to the knowledge base. Be patient and add the next file while the processing of the current one is finished. Otherwise, the browser might crash.

Now that the knowledge base is created, let’s evaluate the responses based on the following questions and answers.

1. on which album was “adam raised a cain” originally released?
   The answer is “Darkness on the Edge of Town”.
2. what is the highest chart position of “Greetings from Asbury Park, N.J.” in the US?
   This answer is #60.
3. what is the highest chart position of the album “tracks” in canada?
   The album did not have a chart position in Canada.
4. in which year was “Highway Patrolman” released?
   The answer is 1982.
5. who produced “all or nothin’ at all”?
   The answer is Jon Landau, Chuck Plotkin, Bruce Springsteen and Roy Bittan.

Navigate to the chat window and type a hash (#). A popup window is shown where you can select a knowledge base. Select the brucespringsteen collection.

Now, you can start asking the questions from above.

Question 1

The answer is correct. Also, note that the source where the information is found is mentioned. This is very convenient, as you will notice shortly. At the bottom, some controls are available to copy the response, to read the response aloud, to retrieve some information like token usage, and so on.

Question 2

The response to question 2 is correct.

According to the context provided, the song “Greetings from Asbury Park, N.J.” by Bruce Springsteen had a chart position of 60 in the US.

Question 3

This question is a tricky one because an album called Tracks exists, but also an album called Tracks 18. Besides that, the album Tracks did not chart at all in Canada.

The response is not wrong, but also not entirely correct. However, you can take a look at the sources and verify the response.

Unfortunately, I don’t have information on the specific chart performance of the album “Tracks” in Canada from the provided context. Can you please provide more information or clarify what’s being asked?

Question 4

The response to question 4 is correct.

According to the context provided, “Highway Patrolman” by Bruce Springsteen was released in 1982.

Question 5

The answer to question 5 is not correct.

I’m not aware of any information about a song called “All or Nothin’ at All.” The context only mentions that it is part of the album “Human Touch” by Bruce Springsteen, but it doesn’t specify who produced it.

Take a closer look at the sources. When adding a document to the knowledge base, the document is split into chunks. Because the documents contain tables, the header information is lost in many chunks, and therefore, the meaning of each cell is lost. That is probably why the correct chunk scores lower than 50%, and the model thinks this information is not relevant.

Considering all responses, the result is certainly not bad because tables are quite challenging to use in RAG. The results in the previous blog were better, but there custom chunking was applied. Do note that in both cases the same model has been used.

Web Search

Another way to enrich the prompt with extra information is to add web pages. Create a new chat window, and just like adding documents, you can add a web page by typing a hash (#) and typing or pasting a web URL. Again, a popup window is shown, where you can select and add the web page. This works similarly to the knowledge you saw earlier.

Add a previous blog: https://dzone.com/articles/why-mvn-install-may-risk-your-builds.

Prompt to create a summary.

The response is not correct. The blog is about Maven, but the summary is entirely wrong.

Response

Based on the provided context, it appears that the user is asking me to summarize the output of a Maven build process.

From the context, it seems like there were two builds executed: one that was successful and another that failed due to dependency issues. The successful build took around 1 second to complete, while the failed build took significantly less time (0.175 seconds). The failed build had dependency issues, as indicated by the “FAILURE” message.

However, without more specific information about what the user wants me to summarize (e.g., specific errors or metrics), it’s difficult to provide a more detailed summary.

Can you please clarify what kind of summary you’re looking for? Are there any specific aspects of the build output you’d like me to highlight?

Second attempt with the following prompt: Create a summary of the added blog post.

Again, the model does not seem to understand what is asked for.

Response

Unfortunately, I don’t have enough information about the content of the added blog post. The provided context appears to be from a developer’s blog or website, but it doesn’t include any details about the actual blog post being referred to.

Could you please provide more context or information about what the blog post is about? Is it related to Maven, Java development, or something else entirely? I’ll do my best to summarize the content based on your query.

The third attempt is to explicitly add the link in the prompt (and add please :-)): 

Please create a summary of the blog https://mydeveloperplanet.com/2024/11/20/why-mvn-install-may-risk-your-builds/.

This response is not entirely correct, but it is not wrong either. However, adding the link to the prompt did make a difference. In order to get a better result, you could change the underlying model. Anyway, Open WebUI did the job well.

Response

The blog post “Why ‘mvn install’ May Risk Your Builds” discusses potential risks associated with using the mvn install command in Maven build processes.

According to the author, mvn install can cause issues when used on feature branches or branches that are not intended for production. This is because mvn install downloads and installs dependencies from the repository into the local Maven cache, which can lead to:

1. Unfinished work being promoted to a more stable branch without proper testing.
   
2. Dependencies from unstable features being installed in production environments.
   

The author recommends using mvn verify instead of mvn install, as it verifies the build and checks for any dependencies issues before proceeding.

Additionally, the blog post discusses the importance of considering separate stages in a build pipeline and using commands like rm -r ~/.m2/repository/com/mydeveloperplanet/mymavenverifyinstallplanet/ to clean up the local Maven repository between builds.

Overall, the author aims to highlight the potential risks of relying on mvn install in certain situations and provide guidance on how to mitigate these risks.

Knowledge PDF

You can also add PDF documents to the knowledge base. Let’s find out whether information from a PDF can be retrieved.

Again, information about Bruce Springsteen will be prompted. The following prompt will be used. The expected response is shown also.

Who was Bruce Springsteen married to?
Bruce Springsteen has been married to Julianne Phillips and to Patti Scialfa.

First, let’s ask llama3.2 this question to see what the model knows already. The answer is partially correct.

• Patti Scialfa is mentioned, but not Julianne Phillips.
• The names of the children are correct, but none of the dates are correct.

Response

Bruce Springsteen has been married to Patti Scialfa since October 6, 1991. They have three children together: Evan James Springsteen (born March 25, 1990), Jessica Rae Springsteen (born April 30, 1992), and Sam Ryan Springsteen (born August 5, 2003).

The information about his personal life can be found on the Bruce Springsteen Wikipedia page. The page is downloaded as a PDF and added to a new knowledge base, brucewiki. Prompt the question while adding this knowledge base.

The answer is not better.

• He has been married twice, but Julianne Phillips is not mentioned again.
• Patti Scialfa was his second wife, not his first.
• The name of the second wife is made up.
• The names of the other ladies are correct.

Response

According to various sources, Bruce Springsteen has been married twice:

1. his first wife, Patti Scialfa, whom he married in 1997 and divorced in 2009
   
2. his second wife, actress Ponselle, but this information is not found in the provided context.
   

However, it’s worth noting that Bruce Springsteen also dated photographer Lynn Goldsmith and model Karen Darvin, as well as actress Joyce Hyser, in the past.

If you’re looking for more specific information on his marriages or relationships, I’d be happy to try and help further.

Just for fun, let’s see what a web search returns as a response. Add the Wikipedia link and use the following prompt: 

Who was Bruce Springsteen married to, use the information from https://en.wikipedia.org/wiki/Bruce_Springsteen.

The response is getting better.

• Springsteen did not divorce from Patti Scialfa, though.
• This time, Julianne Phillips is mentioned.
• The dates are wrong again.

Response

According to Wikipedia, Bruce Springsteen was married to Patti Scialfa from 1997 to 2015 [1]. They were together for over 18 years before their divorce. Before marrying Patti, he was married to Julianne Phillips from 1983 to 1989.

It’s worth noting that his ex-wife Patti Scialfa has been a significant influence on his music and career, and has also pursued a successful music career in her own right.

Controls

At the top right corner, you can set some Controls. The System Prompt and Temperature are typically items you would like to change and control now and then. Be aware that you have this possibility.

Admin Panel

When you are an admin user, you can click your account, and you will be able to select the Admin Panel. You can add roles, set up LDAP, and many other administration tasks can be configured here.

Conclusion

Open WebUI is a very feature-rich UI that works entirely offline. As seen in the previous examples, using RAG has some issues, but this has nothing to do with Open WebUI. You can add custom logic to improve the results using Pipes. Besides that, you can experiment with different models and change parameters in order to improve the results.