I got shut out by my own document search system last February. Not a cloud outage or a bad deploy — I typed a legal clause number I knew was in the contract repository, and the system returned zero results. The query was exact. The document existed. And a keyword search that had indexed every word in the corpus couldn’t find it because the clause used “indemnification” and I’d typed “hold harmless.”
That’s the moment this RAG system started.
The central problem with retrieval-augmented generation — RAG, the pattern where you feed an AI model relevant documents alongside a user’s question so it can answer from real sources instead of hallucinating — isn’t about model quality. It’s about retrieval quality. If the documents you pull back are wrong, the best model in the world will still give you a convincing wrong answer. And enterprise documents are uniquely hostile to retrieval: legal clauses, product codes, regulatory citations, and nested section hierarchies all break the vector-similarity approach that works fine for “how do I reset my password.”
The fix isn’t one strategy. It’s three, fused.
The thing I got wrong first
My first pass treated every document as flat text. The unstructured library pulled paragraphs out of PDFs and I shoved them into a vector database — pgvector, a Postgres extension that stores embeddings (numerical fingerprints of meaning) and lets you search by similarity rather than exact keyword match.
It worked great on the demo. It failed on real documents.
A 90-page services agreement has structure that carries meaning. Section 4.2(a) nested under Article IV modifies the definition in Section 1.1, and if you flatten that hierarchy into semantically-similar chunks, you lose the modification chain. A system that returns the most “relevant” paragraph by embedding distance will happily return Section 1.1’s definition without Section 4.2(a)‘s override, because both paragraphs talk about the same thing.
So I added layout awareness to the ingestion pipeline: coordinate extraction, font-weight analysis, heading-level detection, and visual-emphasis scoring. The processor now tracks where each chunk sits in the document’s structural tree, not just what it says.
Getting the GPU through the whole job
The processing pipeline hits thousands of pages. Doing that on CPU means you walk away and come back tomorrow. Doing it on GPU means you walk away and come back in ten minutes — unless the GPU runs out of memory on page 847 of 900 and you re-run from scratch.
I burned a Saturday on that exact failure mode.
The fix was three things running together: smart batching that watches VRAM pressure and shrinks the batch before it OOMs, automatic GPU selection that chooses the card with the most free memory instead of a hard-coded device ID, and a checkpoint system that writes progress markers to disk so a crash on page 847 means restarting at page 800, not page 1.
# Example of our GPU-accelerated document processor
class DocumentProcessor:
def process_file(self, file_path: str, timeout_seconds: int) -> Tuple[List[Document], Dict]:
with timeout(timeout_seconds):
# GPU-accelerated processing
pdf_elements = self._extract_pdf_elements(file_path)
documents = self._process_elements(pdf_elements)
return documents, self._get_processing_stats()The multi-GPU support came later, after we realized a single A100 was bottlenecked on PDF parsing while the second card sat idle. Now both cards chew through separate files in parallel.
Why one search method was not enough
why hybrid retrieval works — and how to wire it give me the detail
The core insight: dense vector search (embedding similarity) and sparse keyword search (BM25) fail in opposite directions. Vectors miss exact product codes, legal clause numbers, and rare proper nouns — the very tokens that matter most in enterprise docs. BM25 misses paraphrase and synonym matches. Combining them with Reciprocal Rank Fusion (RRF) exploits both without requiring a tuned weighting scheme, because RRF is rank-based and robust to score-scale differences.
The real stack behind each strategy:
| Strategy | Tooling |
|---|---|
| Dense vector | pgvector (Postgres extension) + Azure text-embedding-ada-002 or text-embedding-3-large |
| Sparse keyword | BM25 via rank_bm25 or Postgres tsvector / ts_rank — no separate index needed |
| Layout / structure | unstructured library for PDF element extraction (coordinates, font weight, heading level) |
| Eval loop | ragas — measures Answer Relevance, Faithfulness, Context Recall against a golden Q&A set |
Concrete try-it snippet — RRF fusion in pure Python, no extra dependencies:
from collections import defaultdict
def reciprocal_rank_fusion(*ranked_lists, k=60):
"""Fuse N ranked result lists. k=60 is the standard constant."""
scores = defaultdict(float)
for ranked in ranked_lists:
for rank, doc_id in enumerate(ranked, start=1):
scores[doc_id] += 1.0 / (k + rank)
return sorted(scores, key=scores.__getitem__, reverse=True)
# Usage: fuse vector hits with BM25 hits
fused = reciprocal_rank_fusion(vector_hits, bm25_hits, metadata_hits)Drop this into your retrieval layer and run ragas on your eval set before and after — faithfulness typically improves because the keyword leg keeps grounding citations in exact source text.
Without the hybrid approach, you pick your failure mode: vector-only search misses exact matches, while keyword-only search misses paraphrases. RRF gives you both strengths without hand-tuning weights that drift every time the corpus changes. Metadata — document type, date, author, and department — adds a third signal when neither path is confident.
Retrieval can be right and the answer can still be wrong
Even with perfect retrieval, LLMs fabricate. Not maliciously — statistically. If 90% of the training data uses one phrasing, the model will “complete” your answer that way whether or not the source document says it.
The quality pipeline checks every response three ways:
The first check is answer relevance: whether the response addresses the question or got distracted by a high-scoring irrelevant chunk. It is graded with a confidence score, not treated as a binary gate. The second checks every factual claim against the retrieved sources. Unsupported statements are flagged and annotated rather than deleted, so the user can see which sentences are grounded. The third validates sources: citations are checked for accuracy, source-to-source relationships are tracked, and the whole chain goes into an audit trail so compliance can reconstruct which documents fed the answer.
def validate_response(self, response: str, sources: List[Document]) -> bool:
return all([
self.fact_checker.verify(response, sources),
self.hallucination_detector.check(response),
self.relevance_scorer.evaluate(response) > 0.8
])The 0.8 threshold isn’t magic — it’s where we stopped seeing false passes in the eval set. Below that, the system surfaces uncertainty rather than pretending confidence.
Keep the document structure in the chunks
Chunking — splitting documents into searchable pieces — is the least glamorous part of RAG and the place most systems quietly fail. Split too small and you lose context. Split too large and you dilute search precision.
The chunker here preserves three things: hierarchy (Section 4.2(a) stays nested under Section 4.2), relationships (cross-references between sections survive the split), and layout (a paragraph in a warning callout box gets different treatment from body text).
def smart_chunk(self, document: Document) -> List[Chunk]:
return self.chunk_analyzer.split(
document,
preserve_hierarchy=True,
maintain_relationships=True,
respect_layout=True
)The next gaps
Three things are on the near-term roadmap. First, multi-modal processing: enterprise knowledge also lives in charts, tables, and diagrams. A revenue waterfall chart or an org-chart image needs vision models in the same retrieval pipeline, not a separate image-search silo. Second, usage analytics: who searches for what, which queries return zero results, and which retrieved documents are never cited. That tells me whether retrieval is improving or only passing evals. Third, cross-document relationships. The system understands hierarchy inside one document, but not yet that Section 4.2(a) of the Master Services Agreement modifies Schedule B of the Statement of Work signed six months later. Dynamic knowledge graphs that span documents are the next retrieval-quality jump.
This system runs in production now. It’s not perfect — no retrieval system is — but it fails in predictable, auditable ways rather than silently returning confident wrong answers. For organizations sitting on millions of pages of contracts, compliance documents, and technical specifications, that’s the difference between an AI tool you demo and one you actually use.
The code is open-source at github.com/RooseveltAdvisors/enterprise-rag.