A Retrieval-Augmented Generation (RAG) system that ingests PDF documents and answers questions using both text and image context. The pipeline extracts text, raster images, and full-page scans from PDFs; runs OCR and AI captioning on images; builds dual FAISS vector indices; and serves a Streamlit chat UI powered by a Groq LLM backend.
- Overview
- Architecture
- Repository Structure
- File Reference
- Setup
- Running the Pipeline
- Running the App
- Evaluation
- Embedding Visualization
- Environment Variables
- Requirements
Most RAG systems only handle text. This project extends the standard RAG pattern to handle multimodal PDFs — documents that contain diagrams, charts, figures, scanned pages, or other images alongside prose.
Key capabilities:
- Text extraction from PDFs via PyMuPDF, with sliding-window chunking (500 words, 100-word overlap)
- Image extraction with automated quality filtering (size, variance, uniformity checks)
- OCR on extracted images and scanned full pages via Tesseract
- AI captioning of images using BLIP (
Salesforce/blip-image-captioning-base) - Dual FAISS indices — one for text chunks (via
all-MiniLM-L6-v2), one for image chunks (via CLIPopenai/clip-vit-base-patch32) - Hybrid retrieval with late-fusion score combining (configurable text/image weight)
- Three prompt styles: zero-shot, few-shot, chain-of-thought (CoT)
- Groq LLM backend for answer generation (Mixtral or any Groq-supported model)
- Streamlit chat UI with source attribution, PDF preview panel, and conversation export
- BLEU/ROUGE evaluation of answer quality and Precision/Recall/MAP for retrieval quality
PDFs
│
▼
ingest.py ──────────── Extract text → text/
└──────────── Extract images → orig_img/ (filtered by image_filter.py)
└──────────── Scanned pages → page_images/
│
▼
ocr_images.py ──────── Tesseract OCR + BLIP captions → ocr/
│
▼
chunker.py ─────────── Merge text files + OCR JSON → chunks.json
│
▼
embed.py ───────────── all-MiniLM-L6-v2 (text) → faiss_text.bin
└──────────── CLIP ViT-B/32 (images) → faiss_image.bin
└──────────── chunk_embeddings.json (metadata)
│
▼
retriever.py ───────── unified_search(text, image, hybrid)
│
▼
llm_agent.py ───────── compose_prompt → call Groq API → answer + sources
│
▼
frontend.py ────────── Streamlit Chat UI
multimodal-rag/
│
├── src/
│ ├── ingest.py # PDF text + image extraction
│ ├── ocr_images.py # OCR + BLIP captioning on images
│ ├── chunker.py # Sliding-window chunking (text + image fusion)
│ ├── embed.py # Embedding generation + FAISS index building
│ ├── retriever.py # Unified text/image/hybrid retrieval
│ ├── llm_agent.py # Prompt composition + Groq LLM calls
│ ├── frontend.py # Streamlit chat UI (main app)
│ ├── app.py # Minimal prototype app (early version)
│ ├── image_filter.py # Image quality filtering utilities
│ ├── visualize.py # UMAP embedding visualization
│ ├── evaluate_RAG.py # End-to-end RAG evaluation (BLEU/ROUGE)
│ └── evaluate_retriever.py # Retrieval evaluation (Precision/Recall/MAP)
│
├── data/ # (not committed — create locally)
│ ├── *.pdf # Place your input PDFs here
│ └── extracted/ # Auto-generated pipeline outputs
│ ├── text/ # Per-page plain text files
│ ├── orig_img/ # Raster images extracted from PDFs
│ ├── page_images/ # Full-page images (for scanned/image-heavy pages)
│ ├── ocr/ # OCR + caption JSON per image
│ ├── image_meta/ # Image metadata JSON files
│ ├── chunks.json # All chunks with content and embeddings
│ ├── chunk_embeddings.json # Embedding metadata (text + image indices)
│ ├── faiss_text.bin # FAISS index for text embeddings
│ └── faiss_image.bin # FAISS index for image embeddings
│
├── embedding_visualization.html # Interactive UMAP plot (pre-generated)
├── Multimodal_RAG_Report.pdf # Project report
├── .env # API keys (not committed)
└── requirements.txt # Python dependencies
Processes every PDF in data/. For each page it: (1) extracts raw text and saves it to data/extracted/text/; (2) extracts embedded raster images, runs them through image_filter.py to discard low-quality ones, saves keepers to orig_img/, and writes metadata JSON to image_meta/; (3) if a page has fewer than 80 characters of text (i.e. it's likely scanned), renders the full page as a PNG into page_images/ for downstream OCR.
Iterates over all images in orig_img/ and page_images/. Runs Tesseract OCR on every image to extract embedded text. For raster images (non-full-page), also runs BLIP (Salesforce/blip-image-captioning-base) to generate a natural-language caption. Combines OCR text and caption, merges with existing metadata, and saves a *_processed.json file per image to ocr/.
Note: The Tesseract executable path is currently hardcoded for Windows (
C:\Program Files\Tesseract-OCR\tesseract.exe). Update this line inocr_images.pyfor Linux/macOS.
Aggregates all processed content into a single chunks.json file. Text pages are split using a sliding window (500 words per chunk, 100-word overlap). Image chunks use the combined OCR text and BLIP caption as their content field. Each chunk carries a unique chunk_id, doc_id, page, type (text, image, or page_as_image), and image_path where applicable.
Quality gate for extracted images. Discards images that are: too small (under 50×50 px), nearly blank (grayscale variance below threshold), or nearly uniform in colour (>75% of pixels identical). Called automatically by ingest.py during extraction.
Loads chunks.json and generates embeddings using two models:
- Text chunks →
all-MiniLM-L6-v2(SentenceTransformers) — 384-dimensional embeddings for every chunk - Image chunks →
openai/clip-vit-base-patch32— 512-dimensional visual embeddings for raster images only
Both sets are L2-normalised and stored in separate FAISS IndexFlatIP (cosine similarity) indices saved as faiss_text.bin and faiss_image.bin. Metadata is written to chunk_embeddings.json.
Core retrieval module. Exposes three retrieval modes via unified_search():
"text"— encodes the query withall-MiniLM-L6-v2and searchesfaiss_text.bin"image"— encodes an uploaded image with CLIP and searchesfaiss_image.bin"hybrid"— runs both searches, normalises scores independently, and combines them with a configurable weighted sum (default: 0.6 text + 0.4 image)
Also runnable as a CLI:
python retriever.py --text "your query" --topk 5 --mode hybridOrchestrates the full RAG generation step. Given a query and/or image path, calls unified_search, composes a prompt from the retrieved chunks (with source tags [doc: ..., page: ...]), and calls the Groq API. Supports three prompt templates:
zero_shot— direct question + contextfew_shot— includes a worked example before the questioncot(chain-of-thought) — prompts the model to reason step-by-step before answering
All settings (API URL, key, model) are read from .env.
The main Streamlit application. Features:
- Sidebar controls for prompt style, retrieval mode, top-K, and text/image weights
- Optional image upload for multimodal queries
- Chat history with per-message source attribution (expandable "Retrieved Chunks" panel)
- "View PDF" button per source chunk opens an inline PDF preview in the sidebar
- Conversation export as JSON
An early-stage minimal prototype of the Streamlit app. Kept for reference; frontend.py is the current production UI.
End-to-end evaluation of the full RAG pipeline. Runs a fixed set of test prompts through all three prompt styles (zero_shot, few_shot, cot) and computes BLEU (via SacreBLEU) and ROUGE-1/ROUGE-L against ground-truth reference answers. Saves results to data/extracted/eval_results.json, eval_summary.csv, and eval_outputs.csv.
Evaluates the retrieval component independently using a manually annotated ground-truth mapping of queries to relevant chunk IDs. Computes Precision@5, Recall@5, Average Precision, and MAP across all test queries.
Generates an interactive UMAP scatter plot of all text and image embeddings, coloured by source document and shaped by chunk type (circle = text, square = image). Hovering over a point shows the chunk ID, document, page, snippet, and a thumbnail for image chunks. Output is a self-contained HTML file (embedding_visualization.html).
git clone https://github.com/your-username/multimodal-rag.git
cd multimodal-rag
pip install -r requirements.txt- Ubuntu/Debian:
sudo apt install tesseract-ocr - macOS:
brew install tesseract - Windows: Download from github.com/UB-Mannheim/tesseract and update the hardcoded path in
src/ocr_images.py
Create a .env file in the project root:
GROQ_API_URL=https://api.groq.com/openai/v1/chat/completions
GROQ_API_KEY=your_groq_api_key_here
GROQ_MODEL=mixtral-8x7b-32768Get a free API key at console.groq.com. You can verify connectivity before running the full pipeline:
cd src
python test_api.pymkdir -p data
cp /path/to/your/documents/*.pdf data/.env
data/
__pycache__/
*.pycRun each step from inside the src/ directory:
cd src
# Step 1: Extract text and images from all PDFs
python ingest.py
# Step 2: Run OCR + BLIP captioning on all extracted images
python ocr_images.py
# Step 3: Chunk all text and image content into chunks.json
python chunker.py
# Step 4: Generate embeddings and build FAISS indices
python embed.pyAfter step 4, data/extracted/ will contain chunks.json, chunk_embeddings.json, faiss_text.bin, and (if images were found) faiss_image.bin. The pipeline only needs to be re-run when you add or change PDFs.
cd src
streamlit run frontend.pyOpen http://localhost:8501 in your browser.
Usage:
- Type a question in the chat input, upload an image, or both
- Adjust retrieval mode, prompt style, and text/image weights in the sidebar
- Expand "Retrieved Chunks & Sources" under any answer to inspect what the model used
- Click "View PDF" next to any source chunk to open the source document inline
cd src
python evaluate_RAG.pyEdit the PROMPTS and REFERENCES dictionaries at the top of the file to add your own test questions and ground-truth answers. Results are written to three files in data/extracted/: eval_results.json (full outputs), eval_summary.csv (metric scores), and eval_outputs.csv (generated answers for manual review).
cd src
python evaluate_retriever.pyEdit the ground_truth dictionary in the file to map queries to lists of known-relevant chunk_id values. Prints Precision@5, Recall@5, Average Precision, and MAP to stdout.
To regenerate the interactive UMAP plot after processing new documents:
cd src
python visualize.pyThis overwrites embedding_visualization.html in the project root. Open it in any browser — no server required. Circles = text chunks, squares = image chunks, colour = source document.
A pre-generated version is included in the repo for reference.
| Variable | Description |
|---|---|
GROQ_API_URL |
Groq chat completions endpoint (https://api.groq.com/openai/v1/chat/completions) |
GROQ_API_KEY |
Your Groq API key |
GROQ_MODEL |
Model name (e.g. mixtral-8x7b-32768, llama3-8b-8192) |
Key dependencies (see requirements.txt for the full list):
| Package | Purpose |
|---|---|
pymupdf (fitz) |
PDF parsing and image extraction |
pytesseract |
OCR on images |
transformers |
BLIP captioning, CLIP embeddings |
sentence-transformers |
Text embeddings (all-MiniLM-L6-v2) |
faiss-cpu |
Vector similarity search |
streamlit |
Chat UI |
streamlit-pdf-viewer |
Inline PDF preview panel |
umap-learn |
Dimensionality reduction for visualization |
plotly |
Interactive embedding visualization |
sacrebleu, rouge-score |
Answer quality metrics |
python-dotenv |
.env file loading |
torch |
Model inference |
Pillow |
Image processing |
numpy, pandas |
Data handling |
BLIP and CLIP models are downloaded from HuggingFace Hub on first run (~1 GB total). Subsequent runs use the local cache. The pipeline runs on CPU by default; a CUDA GPU will significantly speed up the OCR captioning and embedding steps.