Local AI · PDF processing · Intermediate
Local RAG for PDFs: Document Search and Answers with a Local LLM

Build a small question-answering system that reads PDF files locally, retrieves relevant passages, answers with a local model, and preserves the page numbers needed to verify each answer.
A useful document assistant should do more than produce a fluent paragraph. It should show which document and page support the answer, avoid sending private files to a third-party service, and fail clearly when the evidence is missing. This article builds that narrow system from first principles.
The design uses retrieval-augmented generation (RAG): extract text from PDFs, split it into searchable chunks, convert the chunks into embeddings, retrieve the nearest chunks with FAISS, and pass those chunks to a local large language model (LLM). Each chunk keeps its source filename and page number throughout the pipeline.
What we are building
The finished program has two commands:
indexreads PDFs from a directory and creates a local vector index.askretrieves relevant passages and asks the local model to answer only from those passages.
The data flow is deliberately simple:
PDF files
│
├─ text extraction
│
├─ chunks with {source, page, text}
│
├─ embedding model
│
├─ FAISS index + metadata.json
│
question
│
├─ question embedding
│
├─ nearest-neighbor search
│
├─ evidence prompt
│
local LLM answer + [source, page] references
This is a minimal baseline, not a complete production document platform. It does not perform OCR, table reconstruction, access control, incremental indexing, or automatic fact checking.
Prerequisites and project layout
Use Python 3.10 or newer if possible. The example relies on PyMuPDF for text extraction, Sentence Transformers for embeddings, FAISS for vector search, and an OpenAI-compatible local HTTP endpoint for generation. The endpoint can be provided by a local model runner such as Ollama, llama.cpp server, or another compatible service; the exact runner is outside this article.
Create a project with this layout:
local-pdf-rag/
├── documents/
├── data/
├── rag.py
└── requirements.txt
Put only documents you are allowed to process in documents/. The example does not upload files anywhere.
Install the Python dependencies
python -m venv .venv
source .venv/bin/activate # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
Use this requirements.txt:
pymupdf
sentence-transformers
faiss-cpu
numpy
requests
The embedding model may download its model files the first time it is loaded. That download is separate from document processing. For a fully offline setup, obtain the model files in advance and point EMBEDDING_MODEL to a local directory.
Step 1: Extract PDF text without losing pages
A PDF is a page-oriented format, so page identity should be captured at extraction time. Do not concatenate the whole document into one string and try to recover page boundaries later. The following function creates one record per page and stores a one-based page number for readers.
from pathlib import Path
import fitz
def extract_pages(pdf_dir: str) -> list[dict]:
records = []
for pdf_path in sorted(Path(pdf_dir).glob("*.pdf")):
with fitz.open(pdf_path) as document:
for page_number, page in enumerate(document, start=1):
text = page.get_text("text").strip()
if text:
records.append({
"source": pdf_path.name,
"page": page_number,
"text": text,
})
return records
Empty pages are skipped. That is useful for ordinary text PDFs, but it also reveals the first important limitation: a scanned PDF may contain only images. PyMuPDF will not magically recognize text inside an image. Such files need an OCR step before indexing.
Step 2: Split pages into searchable chunks
Retrieval works better with focused passages than with entire long pages. We will use character-based chunking with overlap. The overlap keeps context that falls near a boundary.
def chunk_records(
page_records: list[dict],
chunk_size: int = 1200,
overlap: int = 200,
) -> list[dict]:
if overlap >= chunk_size:
raise ValueError("overlap must be smaller than chunk_size")
chunks = []
for record in page_records:
text = " ".join(record["text"].split())
start = 0
while start < len(text):
end = min(start + chunk_size, len(text))
chunk_text = text[start:end].strip()
if chunk_text:
chunks.append({
"source": record["source"],
"page": record["page"],
"text": chunk_text,
})
if end == len(text):
break
start = end - overlap
return chunks
The values here are starting points, not universal settings. A document with dense legal clauses may need smaller chunks. A technical manual with long procedures may benefit from larger chunks. The page number remains the same for every chunk created from that page.
Step 3: Build a FAISS index
An embedding model maps text to vectors. Similar meanings should be near each other in vector space, allowing a question to retrieve passages even when it does not use the same words as the document.
The index below uses cosine similarity by normalizing vectors and then using an inner-product FAISS index. The metadata is stored separately because the vector index contains numbers, not human-readable source information.
import json
from pathlib import Path
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
def build_index(
chunks: list[dict],
output_dir: str = "data",
) -> None:
if not chunks:
raise ValueError("No text chunks were created")
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
model = SentenceTransformer(EMBEDDING_MODEL)
texts = [chunk["text"] for chunk in chunks]
vectors = model.encode(
texts,
convert_to_numpy=True,
normalize_embeddings=True,
show_progress_bar=True,
).astype("float32")
dimension = vectors.shape[1]
index = faiss.IndexFlatIP(dimension)
index.add(vectors)
faiss.write_index(index, str(output_path / "documents.faiss"))
with (output_path / "metadata.json").open("w", encoding="utf-8") as file:
json.dump(chunks, file, ensure_ascii=False, indent=2)
IndexFlatIP performs an exact search over the stored vectors. It is easy to understand and suitable for a small collection. Larger collections may need a different FAISS index, but changing the index type introduces tuning and recall trade-offs that are outside this minimal example.
Step 4: Retrieve evidence for a question
At question time, embed the question with the same model, search for the nearest chunks, and keep their metadata. The same embedding model must be used for documents and questions; otherwise, the vectors are not comparable in the intended way.
def load_index(
index_dir: str = "data",
) -> tuple[faiss.Index, list[dict]]:
path = Path(index_dir)
index = faiss.read_index(str(path / "documents.faiss"))
with (path / "metadata.json").open(encoding="utf-8") as file:
metadata = json.load(file)
if index.ntotal != len(metadata):
raise ValueError("Index and metadata contain different numbers of records")
return index, metadata
def retrieve(
question: str,
index,
metadata: list[dict],
model: SentenceTransformer,
top_k: int = 5,
) -> list[dict]:
query_vector = model.encode(
[question],
convert_to_numpy=True,
normalize_embeddings=True,
).astype("float32")
scores, positions = index.search(query_vector, top_k)
results = []
for score, position in zip(scores[0], positions[0]):
if position < 0:
continue
item = dict(metadata[position])
item["score"] = float(score)
results.append(item)
return results
Similarity scores are useful for sorting, but they are not proof that a passage answers the question. A high-scoring chunk can still be irrelevant. The generator must receive the actual text and the source metadata, not only the score.
Step 5: Ask a local LLM to answer from the evidence
The generation layer below expects an OpenAI-compatible local endpoint. The default URL is an example configuration. Set it to the server you actually run and keep the model name aligned with that server.
import os
import requests
LLM_BASE_URL = os.getenv(
"LLM_BASE_URL",
"http://127.0.0.1:11434/v1",
)
LLM_MODEL = os.getenv("LLM_MODEL", "local-model")
def format_context(results: list[dict]) -> str:
blocks = []
for number, item in enumerate(results, start=1):
blocks.append(
f"[{number}] {item['source']}, page {item['page']}\\n"
f"{item['text']}"
)
return "\\n\\n".join(blocks)
def answer_with_local_llm(
question: str,
results: list[dict],
) -> str:
if not results:
return "I could not find supporting passages in the indexed documents."
context = format_context(results)
system_prompt = (
"You answer questions using only the supplied document excerpts. "
"If the excerpts do not support an answer, say that the answer is "
"not established by the indexed documents. Do not invent facts. "
"Cite supporting excerpts using [source, page N]."
)
payload = {
"model": LLM_MODEL,
"temperature": 0,
"messages": [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": (
f"Question: {question}\\n\\n"
f"Document excerpts:\\n{context}"
),
},
],
}
response = requests.post(
f"{LLM_BASE_URL.rstrip('/')}/chat/completions",
json=payload,
timeout=120,
)
response.raise_for_status()
body = response.json()
return body["choices"][0]["message"]["content"]
The prompt is a guardrail, not a guarantee. A local model can still misunderstand a passage, cite the wrong excerpt, or answer confidently when the evidence is weak. The application should display the retrieved passages beside the answer so a person can inspect them.
Step 6: Connect the pipeline with a small command-line interface
Put the following code after the functions above in rag.py. It provides the two commands described earlier.
import argparse
def run_index(args) -> None:
pages = extract_pages(args.documents)
chunks = chunk_records(
pages,
chunk_size=args.chunk_size,
overlap=args.overlap,
)
build_index(chunks, args.index_dir)
print(f"Extracted pages: {len(pages)}")
print(f"Created chunks: {len(chunks)}")
print(f"Index directory: {args.index_dir}")
def run_question(args) -> None:
index, metadata = load_index(args.index_dir)
model = SentenceTransformer(EMBEDDING_MODEL)
results = retrieve(args.question, index, metadata, model, args.top_k)
answer = answer_with_local_llm(args.question, results)
print("\\nAnswer:\\n")
print(answer)
print("\\nRetrieved evidence:\\n")
for item in results:
print(
f"- {item['source']}, page {item['page']} "
f"(score={item['score']:.4f})"
)
print(f" {item['text'][:300]}...")
def main() -> None:
parser = argparse.ArgumentParser(
description="Search local PDFs and answer with a local LLM"
)
subparsers = parser.add_subparsers(required=True)
index_parser = subparsers.add_parser("index")
index_parser.add_argument("--documents", default="documents")
index_parser.add_argument("--index-dir", default="data")
index_parser.add_argument("--chunk-size", type=int, default=1200)
index_parser.add_argument("--overlap", type=int, default=200)
index_parser.set_defaults(function=run_index)
ask_parser = subparsers.add_parser("ask")
ask_parser.add_argument("question")
ask_parser.add_argument("--index-dir", default="data")
ask_parser.add_argument("--top-k", type=int, default=5)
ask_parser.set_defaults(function=run_question)
args = parser.parse_args()
args.function(args)
if __name__ == "__main__":
main()
Run the pipeline
First create the index:
python rag.py index
A successful run should create data/documents.faiss and data/metadata.json. The printed page and chunk counts are generated by your local run; they depend on the PDFs you supplied.
Then configure the local endpoint and ask a question:
export LLM_BASE_URL="http://127.0.0.1:11434/v1"
export LLM_MODEL="your-local-chat-model"
python rag.py ask "What is the document's stated retention period?"
On Windows PowerShell, use:
$env:LLM_BASE_URL = "http://127.0.0.1:11434/v1"
$env:LLM_MODEL = "your-local-chat-model"
python rag.py ask "What is the document's stated retention period?"
The expected shape of the response is illustrative:
Answer:
The indexed excerpts state that the retention period is ...
[policy.pdf, page 4]
Retrieved evidence:
- policy.pdf, page 4 (score=0.XXXX)
The relevant passage begins ...
The exact wording, score, and answer depend on the documents, embedding model, local LLM, and endpoint implementation. They are not fixed test results.
Verification checklist
Before trusting the assistant with a real document set, verify the pipeline with questions whose answers you already know.
- Index a PDF with a distinctive sentence on a known page.
- Ask a question using different wording from the sentence.
- Confirm that retrieval lists the expected filename and one-based page number.
- Open the cited page and check that it actually supports the answer.
- Ask about a fact that does not appear in the documents.
- Confirm that the model says the evidence is insufficient instead of filling the gap.
- Index an empty directory and confirm that the program fails with a clear “No text chunks were created” error.
- Change or remove one metadata record and confirm that the index/metadata consistency check catches the mismatch.
These are verification actions for you to perform. This article does not claim that they were run in the current workspace.
Failure cases and practical fixes
The PDF produces no text
The document may be scanned, encrypted, or structured in a way that the extractor does not expose as ordinary text. Add OCR before the extraction stage, or use a PDF with a real text layer. OCR output should be reviewed because recognition errors can change names, numbers, and page content.
Answers cite the wrong page
Check whether your extraction library uses zero-based page numbers. This example converts the page counter to one-based numbering with enumerate(..., start=1). Also check that the metadata list has not been reordered independently of the FAISS vectors.
Retrieval misses an obvious passage
Try a different embedding model, increase top_k, adjust chunk size, and inspect the extracted text. Headers, footers, broken columns, and unusual terminology can make semantic retrieval less effective. A lexical search fallback can be useful for exact identifiers, policy codes, and numbers.
The model answers from general knowledge
Strengthen the system prompt, lower temperature, show the retrieved context in the user interface, and require citations. Most importantly, treat the answer as a summary of retrieved evidence rather than as an independent source of truth. If the model repeatedly ignores the evidence, change the model or add an answer validation step.
Memory or speed is poor
The embedding model and the LLM have different resource requirements. Reduce batch sizes, use a smaller local model, index fewer chunks at a time, or choose a quantized model supported by your local runner. Exact FAISS search is straightforward but scales linearly with the number of stored vectors.
Privacy and operational boundaries
Keeping the PDF and generation endpoint on the same machine can reduce external exposure, but “local” does not automatically mean secure. Check where model downloads, logs, temporary files, telemetry, backups, and crash reports are written. Restrict access to the HTTP endpoint and protect the indexed directory like the original documents.
Do not place passwords, API keys, personal identifiers, or confidential examples in source code or shell history. If a model runner requires authentication, use its documented secret-management mechanism rather than hard-coding a credential in rag.py.
Limitations and next steps
This baseline is intentionally small. It preserves page references for text extraction, but it does not guarantee perfect citations. It may struggle with scanned documents, multi-column layouts, tables, figures, footnotes, very long answers, and questions requiring information spread across many pages.
Reasonable next improvements are:
- Add OCR for image-only PDFs.
- Store page coordinates and text blocks for more precise citations.
- Add a lexical or hybrid retriever for exact terms and identifiers.
- Use reranking before sending context to the LLM.
- Persist document hashes so unchanged PDFs are not reindexed.
- Expose retrieved excerpts in the interface alongside the generated answer.
- Add automated evaluation questions with known source pages.
The central design choice should remain unchanged: document text and its source metadata travel together from extraction through retrieval to the final answer. That is what makes a local PDF assistant inspectable rather than merely fluent.
Summary
A minimal local PDF RAG system needs four durable pieces: page-aware extraction, chunks with metadata, vector retrieval, and a local generation endpoint. FAISS finds candidate passages; the LLM turns those passages into an answer; the source and page fields let a reader verify the result. The system is useful when it clearly separates retrieved evidence from generated language and openly reports when the indexed documents do not contain an answer.
Continue with the guides or look up related terms in the glossary.