Building a Multimodal RAG Pipeline with NVIDIA NeMo Retriever, Hosted NIMs, LanceDB, Reranking, and Grounded Generation

Introduction

In this tutorial, you'll build a multimodal Retrieval-Augmented Generation (RAG) pipeline using NVIDIA NeMo Retriever, hosted NIMs, LanceDB, reranking, and grounded generation. This pipeline enables you to process PDF documents, extract text and images, and use them to generate contextually relevant answers to user queries. The tutorial assumes you have a basic understanding of Python, RAG concepts, and machine learning pipelines.

By the end of this tutorial, you will have a working multimodal RAG system that can:

• Extract text and images from PDFs
• Embed documents using NVIDIA's NeMo Retriever
• Store embeddings in LanceDB for fast retrieval
• Perform reranking to improve relevance
• Generate grounded answers using hosted NIMs

Prerequisites

Before beginning, ensure you have the following:

1. Python 3.12 installed
2. Access to a NVIDIA NIM endpoint (can be hosted or using a free trial)
3. Basic understanding of embeddings and vector databases

Step-by-Step Instructions

1. Set Up Your Python Environment

Create a new virtual environment and install required packages.

python -m venv multimodal_rag_env
source multimodal_rag_env/bin/activate  # On Windows: multimodal_rag_env\Scripts\activate
pip install nemo-retriever lancedb torch transformers pdfplumber

Why: This sets up an isolated Python environment with all necessary libraries for our pipeline. NeMo Retriever provides multimodal embedding capabilities, while LanceDB is used for efficient vector storage and retrieval.

2. Extract Text and Images from PDF

Write a script to extract text and images from a PDF file.

import pdfplumber

def extract_pdf_content(pdf_path):
    text_content = []
    image_paths = []
    
    with pdfplumber.open(pdf_path) as pdf:
        for i, page in enumerate(pdf.pages):
            # Extract text
            text = page.extract_text()
            if text:
                text_content.append(text)
            
            # Extract images
            images = page.images
            for img in images:
                img_path = f"page_{i}_img_{len(image_paths)}.png"
                with open(img_path, "wb") as f:
                    f.write(img["stream"].read())
                image_paths.append(img_path)
    
    return "\n".join(text_content), image_paths

Why: PDF extraction is the first step in preparing documents for embedding. This function handles both text and image extraction, which are essential for multimodal RAG.

3. Initialize NeMo Retriever for Embedding

Initialize the NeMo Retriever to create embeddings for the extracted content.

from nemo_retriever import NeMoRetriever

# Initialize the retriever with a multimodal model
retriever = NeMoRetriever(
    model_name="nvidia/nemo-multimodal-embeddings",
    device="cuda"  # Use GPU if available
)

Why: NeMo Retriever provides access to state-of-the-art multimodal models that can generate embeddings for both text and image content, enabling a more comprehensive retrieval system.

4. Store Embeddings in LanceDB

Create a LanceDB table to store document embeddings.

import lancedb
import pandas as pd

# Create LanceDB connection
conn = lancedb.connect("./lancedb")

# Create table
if "documents" not in conn.table_names():
    table = conn.create_table("documents", schema={
        "id": "string",
        "text": "string",
        "image_path": "string",
        "embedding": "vector(1024)"
    })
else:
    table = conn.open_table("documents")

# Prepare data for insertion
embeddings = retriever.embed(texts=text_content, images=image_paths)

# Insert into LanceDB
for i, (text, image_path, embedding) in enumerate(zip(text_content, image_paths, embeddings)):
    table.add([{"id": str(i), "text": text, "image_path": image_path, "embedding": embedding}])

Why: LanceDB is optimized for vector search, making it ideal for storing and retrieving embeddings efficiently. This allows for fast retrieval of relevant documents during query processing.

5. Implement Reranking

Use a reranker to improve the relevance of retrieved documents.

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Initialize reranker
reranker = AutoModelForSequenceClassification.from_pretrained(
    "cross-encoder/ms-marco-MiniLM-L-12-v2"
)

# Rerank documents
def rerank_documents(query, retrieved_docs):
    scores = []
    for doc in retrieved_docs:
        score = reranker(
            tokenizer(query, doc["text"], return_tensors="pt")
        ).logits
        scores.append(score)
    
    # Sort by score
    sorted_docs = sorted(zip(retrieved_docs, scores), key=lambda x: x[1], reverse=True)
    return [doc[0] for doc in sorted_docs]

Why: Reranking improves the relevance of retrieved documents by re-scoring them based on their relevance to the query, which significantly enhances the quality of generated responses.

6. Generate Grounded Answers Using Hosted NIMs

Use the hosted NIM endpoint to generate answers based on the retrieved and reranked documents.

import requests

def generate_answer(query, context):
    payload = {
        "prompt": f"Answer the question based on the provided context: {query} Context: {context}",
        "max_tokens": 512,
        "temperature": 0.5
    }
    
    response = requests.post(
        "https://api.nvidia.com/v1/generation",
        headers={"Authorization": "Bearer YOUR_API_KEY"},
        json=payload
    )
    
    return response.json()["choices"][0]["text"]

Why: Using hosted NIMs allows you to leverage NVIDIA's powerful language models for grounded generation, ensuring that the generated answers are both relevant and factual based on the retrieved context.

7. Putting It All Together

Create a main function to orchestrate the entire pipeline.

def multimodal_rag_pipeline(pdf_path, query):
    # Step 1: Extract content
    text_content, image_paths = extract_pdf_content(pdf_path)
    
    # Step 2: Embed documents
    embeddings = retriever.embed(texts=text_content, images=image_paths)
    
    # Step 3: Store in LanceDB
    # (Implementation from Step 4)
    
    # Step 4: Retrieve documents
    retrieved_docs = table.search(query).limit(5).to_list()
    
    # Step 5: Rerank documents
    reranked_docs = rerank_documents(query, retrieved_docs)
    
    # Step 6: Generate answer
    context = " ".join([doc["text"] for doc in reranked_docs[:3]])
    answer = generate_answer(query, context)
    
    return answer

Why: This function ties together all components of the pipeline, providing a clean interface for processing user queries and returning relevant answers.

Summary

In this tutorial, you built a multimodal RAG pipeline using NVIDIA NeMo Retriever, LanceDB, reranking, and hosted NIMs. You learned how to extract content from PDFs, generate embeddings, store them efficiently in LanceDB, rerank documents, and generate grounded answers using NVIDIA's hosted models. This pipeline is highly scalable and can be extended to handle more complex multimodal content, including videos and audio files.