DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workkloads.

Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Architectural Patterns for Enterprise Generative AI Apps: DSFT, RAG, RAFT, and GraphRAG
  • Pivoting Database Systems Practices to AI: Create Efficient Development and Maintenance Practices With Generative AI
  • Artificial Intelligence, Real Consequences: Balancing Good vs Evil AI [Infographic]
  • AI Meets Vector Databases: Redefining Data Retrieval in the Age of Intelligence

Trending

  • Docker Base Images Demystified: A Practical Guide
  • How Large Tech Companies Architect Resilient Systems for Millions of Users
  • Unlocking AI Coding Assistants Part 4: Generate Spring Boot Application
  • Top Book Picks for Site Reliability Engineers
  1. DZone
  2. Data Engineering
  3. AI/ML
  4. A Framework for Building Semantic Search Applications With Generative AI

A Framework for Building Semantic Search Applications With Generative AI

This article explores a framework for building semantic search applications with generative AI, enabling developers with a unified scalable framework.

By 
Lakshmanan Sethu Sankaranarayanan user avatar
Lakshmanan Sethu Sankaranarayanan
·
Jun. 18, 24 · Tutorial
Likes (3)
Comment
Save
Tweet
Share
3.0K Views

Join the DZone community and get the full member experience.

Join For Free

Search applications play a critical role in this digital era. It enables enterprises to find the right and valuable information at the right time to make informed decisions. The efficient search application proves to improve productivity and efficiency for organizations to thrive. The traditional keyword search applications fail to understand the meaning, intent, and context of users or customers, They don't have the power to understand the nuances of human language. They are heavily built on keyword matching, which may lead to inaccurate results in understanding the intent behind the search. The developers and architects are facing challenges in building search applications that go beyond mere keyword matching. 

Now it has been made it easier and more powerful to reimagine searches by leveraging cutting-edge technologies like natural language processing (NLP), machine learning (ML), and models. These advanced techniques can unlock the ability to truly understand the meaning and context of user queries which leads to more accurate and personalized results.

Why Semantic Search?

Semantic search uses natural language processing (NLP) to understand the intent of the search being made. With the rise of LLM,  generative AI/LLM solutions can be combined with semantic search to capture the full essence of a customer query to provide personalized, accurate, and contextual search results. In this article, we will explore a framework for building semantic search applications with generative AI. This framework will enable developers, cloud architects, and cloud leaders with a unified scalable framework for building semantic search applications.

Below are possible types of semantic search solutions that can be built with Generative AI solutions across various industries. For example:

  1. Personalized recommendation: The search solution can provide products/movies based on user behavior, choices, and earlier search history.
  2. Knowledge management/discovery: It can be used to create searchable content of documents, which results in quick access and discovery.
  3. Image search: The search application can search based on the image queries, which will provide similar images or provide the text results/summary for the given image.
  4. Voice search/recommendations: With the integration of voice recognition libraries, the system can provide recommendations based on voice commands or queries.
  5. Multimodal search: This enables the combined search of text, image, and video together as input to provide summarized results.

Let's take a look at the framework approach of building semantic search applications with Generative AI.
Framework for semantic search applications with Gen AI

Figure 1: Framework for semantic search applications with Gen AI

Let's look at the framework components below. 

1. Knowledge Graph

This is the source data used for semantic search. This can be documents or knowledge graph databases. In this step, we need to ingest data from various sources (documents, databases, APIs, etc.) and preprocess the data by cleaning, normalizing, and removing the duplicates.

2. Chunking of Knowledge Data

In this step, data is chunked if needed. The chunking of data helps improve computational speed to calculate the embeddings of the data, improves memory utilization, and provides accurate search results. It retains the semantic richness of the information which enables more contextual search results. The chunked data can optimize the embedding evaluation process, leading to more accurate representations and improved search capabilities in your application.

S.No Langchain
1 CharaterTextSplitter
2.  RecursiveCharacterTextSplitter

Here is the sample code to split the knowledge graph documents into chunks with Python:

Python
 
def get_split_knowledgegraphdocuments(index_path: str) -> List[str]:
    """
    This function is used to chunk documents 


    Args:
    index_path: Path of the dataset folder containing the documents.


    Returns:
    List of chunked, or split documents.
    """


    split_kdocs = []


    for f_name in os.listdir(index_path):
        print(f" your file_name : {file_name}")
        if fname.endswith(".pdf"):
            loader = UnstructuredPDFLoader(index_path + f_name)
        else:
            loader = TextLoader(index_path + f_name)


        text_splitter = CharacterTextSplitter(chunk_size=8192, chunk_overlap=128)
        split_docs.extend(text_splitter.split_documents(loader.load()))


    return split_kdocs



3. Evaluate the Embedding

An embedding is a technique to represent data points as vectors in a lower-dimensional space. In the real world, most of our text, image, or video data are very high dimensional in nature. This essentially translates complex data, like text or images, into a format that machines can understand and manipulate more easily. Embeddings are a powerful tool for bridging the gap between complex real-world data representation and ML algorithms. This transformational representation helps with building effective and intelligent applications. In this step, evaluate the embedding of the source data with the help of embedding APIs available in the cloud market. 

S.No Google Cloud Azure AWS  Open Source
1 Vertex AI Embeddings Azure Open AI embeddings Amazon Titan Multi-modal embedding model

E5-Mistral-7B-instruct

4. Vector Databases

Vector databases are a valuable tool for unlocking the power of unstructured and semi-structured data. By representing information as vectors and enabling similarity search, they pave the way for applications like semantic search, recommendation systems, and advanced NLP tasks. Vector databases are optimized for unstructured and semi-structured data, enabling searches based on semantic similarity.

S.No Google Cloud Azure  AWS Open Source
1 Vertex AI Vector Search Azure AI Search Amazon Kendra PineCone

Here is the sample code to generate embedding and store it in a vector database with the Google Cloud VertexAI platform.

Python
 
# Create  VertexAI Embeddings objects for the knowledge graph
EMBEDDING_NUM = 5


vembeddings = VertexAIEmbeddings(
    model_name=EMBEDDING_MODEL, batch_size=EMBEDDING_NUM)


split_kdocs = get_split_documents(PATH)


db = Chroma.from_documents(
    documents=split_kdocs, embedding=vembeddings, persist_directory=PERSIST_PATH
)
db.persist() 


5. Semantic/Vector Search 

The semantic/vector search algorithm calculates the search results from the embeddings stored in a vector database for a given user query. The similarity scores between query embeddings and document embeddings are calculated here. The output of this results in semantic search systems that can rank and present relevant results to the user.

6. Ranking and Results

The relevant scoring and ranking are performed, which evaluates the semantic similarity between the query and the retrieved information from the vector database, which considers factors such as context, entities, relationships, and overall query topic relevance.

Then the generated responses or summaries are then ranked based on their relevance scores and the most relevant results will appear at the top. The ranking algorithm produces results that consider user choices, personalization, and history.

7. Generative AI Model Output

Generative AI models are a critical component in semantic search applications to ensure a more natural and intuitive interaction between customers and the knowledge retrieval system. These models leverage their natural language understanding capabilities to interpret user queries accurately, match them semantically with relevant data in the knowledge graph or repository, and generate human-like responses or summaries or translate the search results depending on the user preference. You can use the multimodal capabilities of the foundation model to provide more interactive search, audio, video, and voice search, personalized recommendations, and summary. There is a possibility of fine-tuning the Generative AI model to suit your needs as well.

S.No Google Cloud Azure  AWS  Open Source
1. Gemini Flash 1.5 Azure Open AI Amazon Bedrock Llama

Conclusion

The combination of Generative AI and semantic search opens up a world of possibilities across various industries such as healthcare, E-commerce, finance/banking, and manufacturing for finding relevant information, personalized recommendations, effective summarization multi-language translation, and knowledge discovery.

AI Semantic search Framework Knowledge Graph generative AI vector database

Opinions expressed by DZone contributors are their own.

Related

  • Architectural Patterns for Enterprise Generative AI Apps: DSFT, RAG, RAFT, and GraphRAG
  • Pivoting Database Systems Practices to AI: Create Efficient Development and Maintenance Practices With Generative AI
  • Artificial Intelligence, Real Consequences: Balancing Good vs Evil AI [Infographic]
  • AI Meets Vector Databases: Redefining Data Retrieval in the Age of Intelligence

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!