How Does GenAI Powered Search Engine Work?

Generative Artificial Intelligence (GenAI) technology is setting a new trend in how information is being consumed in this AI-first era. It is built on top of Large Language Models (LLMs), which play a huge role in building an assistive search engine th…


This content originally appeared on DEV Community and was authored by RAGAVI

Generative Artificial Intelligence (GenAI) technology is setting a new trend in how information is being consumed in this AI-first era. It is built on top of Large Language Models (LLMs), which play a huge role in building an assistive search engine that is powered by GenAI capabilities.

The problem with LLMs is that they cannot provide any recent or present information as they need months to retrain with new data. To overcome this limitation, an innovative architecture is proposed that sits on top of LLMs.

What is the Retrieval Augmented Generation (RAG) framework?

The Retrieval Augmented Generation (RAG) is an elegant way to augment recent or new information to be presented to the underlying LLMs such that it can understand the question that seeks new information. The RAG framework powers all the GenAI-based search engines or any search engine that provides context-aware answers to customers’ questions.

Image description

RAG architecture

The RAG architecture consists of a Retriever Module and a Generator Module. For RAG architecture to work, we need to chunk all the knowledge base content into small chunks. There are many ways to chunk all the knowledge base content, such as

  • Chunk them based on content hierarchy
  • Chunk them based on the use case
  • Chunk them based on content type and use case

Once the text data is chunked, then all these chunks need to be converted into text embedding. A plethora of APIs are available from GenAI tool vendors whereby the embedding model is a popular API quickly and cheaply. OpenAI Ada text embedding model is a popular API that is widely used.

The next step in the process is to store all text embeddings along with their related chunks and metadata in a vector database.

Image description

To continue reading about how GenAI powered search engine work? Click here


This content originally appeared on DEV Community and was authored by RAGAVI


Print Share Comment Cite Upload Translate Updates
APA

RAGAVI | Sciencx (2024-06-26T07:31:15+00:00) How Does GenAI Powered Search Engine Work?. Retrieved from https://www.scien.cx/2024/06/26/how-does-genai-powered-search-engine-work/

MLA
" » How Does GenAI Powered Search Engine Work?." RAGAVI | Sciencx - Wednesday June 26, 2024, https://www.scien.cx/2024/06/26/how-does-genai-powered-search-engine-work/
HARVARD
RAGAVI | Sciencx Wednesday June 26, 2024 » How Does GenAI Powered Search Engine Work?., viewed ,<https://www.scien.cx/2024/06/26/how-does-genai-powered-search-engine-work/>
VANCOUVER
RAGAVI | Sciencx - » How Does GenAI Powered Search Engine Work?. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/06/26/how-does-genai-powered-search-engine-work/
CHICAGO
" » How Does GenAI Powered Search Engine Work?." RAGAVI | Sciencx - Accessed . https://www.scien.cx/2024/06/26/how-does-genai-powered-search-engine-work/
IEEE
" » How Does GenAI Powered Search Engine Work?." RAGAVI | Sciencx [Online]. Available: https://www.scien.cx/2024/06/26/how-does-genai-powered-search-engine-work/. [Accessed: ]
rf:citation
» How Does GenAI Powered Search Engine Work? | RAGAVI | Sciencx | https://www.scien.cx/2024/06/26/how-does-genai-powered-search-engine-work/ |

Please log in to upload a file.




There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.