How to Build a RAG App in Python with Ollama and Chroma

Key takeaways

• You can build a RAG app in Python that runs entirely on your own machine with two open-source pieces: Ollama for embeddings and text generation, and Chroma as the vector store.
• The pipeline has four steps: split documents into chunks, turn each chunk into an embedding, store and search those vectors, then pass the best matches to a local LLM with an instruction to answer only from them.
• Use the same embedding model for documents and questions. With nomic-embed-text, add the search_document: and search_query: prefixes its model card asks for.
• Chunk size is a trade-off: small chunks retrieve precisely but lose context; large chunks keep context but blur the match and can be silently truncated by the embedding model.
• Test retrieval on its own before judging answers. A short list of questions with known source passages catches most problems.

To build a RAG app in Python that runs fully offline, you need two open-source pieces: Ollama for embeddings and text generation, and Chroma as the vector store. New to Ollama? See how to run a local LLM with Ollama.

What is a RAG app in Python, and why build one locally?

The term comes from a 2020 paper by Patrick Lewis and colleagues, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", accepted at NeurIPS 2020. The authors paired a generator model with a searchable index of Wikipedia passages so the model could draw on retrieved text instead of relying only on what it memorised during training. They reported that this made answers more specific and factual on open-domain question answering.

Modern RAG apps follow the same idea with simpler tools. Instead of training a retriever, you use a ready-made embedding model that converts text into a list of numbers (a vector). Passages with similar meaning end up with vectors that point in similar directions, so finding relevant text becomes a nearest-neighbour search.

Running the whole stack locally matters for three reasons. Private documents, such as internal policies or research notes, never go to a third-party service. Costs are fixed by your hardware rather than by usage. And the app keeps working offline. The trade-off is that small local models are less capable than the largest hosted ones, which makes good retrieval even more important. If you prefer a hosted model for the generation step, the retrieval code below works unchanged; see our Gemini 4 Argon API tutorial for a cloud example.

The tools and models used in this guide

Ollama exposes embedding models through its /api/embed endpoint and the matching ollama.embed() function in its Python library. Ollama's April 2024 announcement of embedding support listed three models; the table below adds the details published on each model's page.

Documentation checked on 22 October 2026. Package versions change often, so check the linked documentation if a call behaves differently.

Step 1: Install the packages and pull the models

You need a recent Python 3 (the code was written for 3.10 or newer) and a running Ollama install. Create a project folder with a virtual environment, then install the two Python libraries:

mkdir local-rag && cd local-rag
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install ollama chromadb

Next, download one embedding model and one chat model:

ollama pull nomic-embed-text
ollama pull llama3.2

Put the files you want to search in a folder called docs/. This tutorial reads plain .txt and .md files. For PDFs or web pages, convert them to text first and check the output, because messy extraction (broken lines, repeated headers) is a common cause of poor answers.

Step 2: Split documents into chunks

Embedding models work on limited spans of text, and a question usually matches one paragraph rather than a whole file. So the first job is to cut each document into overlapping chunks. The function below aims for about 1,000 characters per chunk, prefers to break at a paragraph or sentence boundary, and repeats 150 characters between neighbouring chunks so an idea that straddles a boundary is not lost. Chunking matters because a model can only read so much at once; see what a context window is.

# ingest.py
import pathlib

import chromadb
import ollama

EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 1000   # characters
OVERLAP = 150       # characters repeated between neighbouring chunks
BATCH = 32          # chunks embedded per request


def chunk_text(text, size=CHUNK_SIZE, overlap=OVERLAP):
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        if end < len(text):
            # Prefer to cut at a paragraph break, then at a sentence end.
            cut = text.rfind("\n\n", start, end)
            if cut == -1:
                cut = text.rfind(". ", start, end)
            if cut > start + size // 2:
                end = cut + 1
        chunks.append(text[start:end].strip())
        if end >= len(text):
            break
        start = end - overlap
    return [c for c in chunks if c]

Counting characters instead of tokens keeps the code dependency-free. A 1,000-character chunk of English prose is a few hundred tokens, comfortably inside the 2K window that the Ollama library lists for nomic-embed-text.

How to choose a chunk size

There is no universal best value; it depends on how your documents are written and what people ask. The table summarises the trade-offs to test against your own data.

One detail is easy to miss. Ollama's /api/embed endpoint truncates input that exceeds the model's context length by default (truncate defaults to true), so an oversized chunk does not raise an error. Its end is simply ignored. If you experiment with very large chunks, pass truncate=False so that overflow fails loudly.

Step 3: Create embeddings with Ollama and store them in Chroma

Now add the embedding and storage code to ingest.py. Two details matter. First, the nomic-embed-text model card says inputs must start with a task prefix: search_document: for text being indexed and search_query: for questions. Second, Chroma's default distance is squared L2; this guide switches the collection to cosine distance, which compares only the direction of vectors.

def embed(texts, prefix):
    # ollama.embed accepts a list and returns one vector per input.
    resp = ollama.embed(model=EMBED_MODEL, input=[prefix + t for t in texts])
    return resp["embeddings"]


def main():
    client = chromadb.PersistentClient(path="./rag_db")
    collection = client.get_or_create_collection(
        name="docs",
        configuration={"hnsw": {"space": "cosine"}},
    )
    for path in sorted(pathlib.Path("docs").rglob("*")):
        if path.suffix.lower() not in {".txt", ".md"}:
            continue
        chunks = chunk_text(path.read_text(encoding="utf-8"))
        for i in range(0, len(chunks), BATCH):
            batch = chunks[i:i + BATCH]
            collection.upsert(
                ids=[f"{path}-{i + j}" for j in range(len(batch))],
                documents=batch,
                embeddings=embed(batch, "search_document: "),
                metadatas=[{"source": str(path), "chunk": i + j}
                           for j in range(len(batch))],
            )
        print(f"indexed {path} ({len(chunks)} chunks)")


if __name__ == "__main__":
    main()

Run it with python ingest.py. PersistentClient writes the database to the rag_db folder and reloads it next time, so you only re-embed when files change. Using upsert with stable IDs means a second run overwrites existing chunks instead of duplicating them. The Chroma getting-started guide recommends the same pattern. If you shorten a file, delete its old chunk IDs too, or rebuild the folder.

We pass our own vectors through embeddings=, so Chroma stores them as given and does not need its built-in embedding function. The documents and metadatas fields keep the original text and its file name next to each vector, which is what makes citations possible later.

Step 4: Retrieve passages and ground the LLM's answer

The query side embeds the question with the same model (now with the search_query: prefix), asks Chroma for the nearest chunks, drops weak matches, and sends the rest to the chat model with numbered labels.

# ask.py
import sys

import chromadb
import ollama

EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "llama3.2"
TOP_K = 4
MAX_DISTANCE = 0.5   # cosine distance; tune on your own data

SYSTEM = (
    "Answer using only the numbered context passages. "
    "Cite passages like [1] or [2] after each claim. "
    "If the passages do not contain the answer, say you don't know."
)

client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_collection(name="docs")


def retrieve(question, k=TOP_K):
    q = ollama.embed(model=EMBED_MODEL, input="search_query: " + question)
    res = collection.query(
        query_embeddings=[q["embeddings"][0]],
        n_results=k,
        include=["documents", "metadatas", "distances"],
    )
    return [
        {"text": doc, "source": meta["source"], "chunk": meta["chunk"], "distance": dist}
        for doc, meta, dist in zip(
            res["documents"][0], res["metadatas"][0], res["distances"][0]
        )
    ]


def answer(question):
    hits = [h for h in retrieve(question) if h["distance"] <= MAX_DISTANCE]
    if not hits:
        return "I could not find this in the documents.", []
    context = "\n\n".join(
        f"[{n}] ({h['source']}, chunk {h['chunk']})\n{h['text']}"
        for n, h in enumerate(hits, 1)
    )
    resp = ollama.chat(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
        options={"temperature": 0},
    )
    return resp["message"]["content"], hits


if __name__ == "__main__":
    text, hits = answer(" ".join(sys.argv[1:]))
    print(text)
    print("\nSources:")
    for n, h in enumerate(hits, 1):
        print(f"[{n}] {h['source']} (chunk {h['chunk']}, distance {h['distance']:.3f})")

Ask a question with python ask.py "What is our refund window?". The script prints the answer followed by the files and chunks it was based on, so a reader can open the original and check.

Three choices keep answers grounded. The system message limits the model to the supplied passages and asks for bracketed citations. A temperature of 0 reduces random wording. And the distance filter means the model is never handed irrelevant text and invited to improvise. Cosine distance in Chroma runs from 0 (same direction) upward, so lower is better. The 0.5 cut-off is only a starting point: print the distances for questions you know the answer to, and set the threshold just above the scores of the correct chunks.

How to evaluate a local RAG app

Most RAG failures are retrieval failures: the right passage never reached the model. Test the two halves separately with a small, hand-made set of 20 to 30 questions whose answers you know and can point to in the source files.

1. Retrieval hit rate. For each question, call retrieve() and check whether the expected file and chunk appear in the top results. If they often do not, look at chunking and text extraction before changing models.
2. Faithfulness. Read each answer next to its cited passages. Every claim should be supported by the text it cites, with no extra facts.
3. Citation accuracy. Check that the bracket numbers point to the passage that actually contains the claim, not just any retrieved passage.
4. Refusal. Include a few questions the documents cannot answer. The app should say it does not know rather than guess.
5. Freshness. Edit a document, re-run ingest.py, and confirm the answer changes. Stale chunks are a common source of wrong answers.
6. Speed. Time ingestion and a typical query on your hardware so you know what users will experience.

Keep the question set in a file and re-run it after every change to chunk size, model, prefix or threshold. That turns tuning into a measurement rather than a guess.

Common problems and fixes

• Relevant text exists but is not retrieved. Check the task prefixes, confirm that documents and questions use the same embedding model, and try smaller chunks or more overlap.
• Results look random after switching models. Vectors from different models are not comparable. Delete rag_db and re-index whenever you change the embedding model.
• Answers ignore the context. Make the system message stricter, reduce TOP_K so the prompt is shorter, or try a larger chat model if your hardware allows.
• Good passages rank below weaker ones. Add a re-ranking step. The Sentence Transformers library documents a "retrieve and re-rank" pattern in which a cross-encoder re-scores the first-stage candidates.
• Connection errors. The Ollama app or ollama serve must be running; the Python library talks to it on localhost:11434.

Where to go next

Once the basic loop works, there are natural upgrades. You can store extra metadata such as dates or departments and filter queries on it. You can stream the chat response for a faster-feeling interface. Or you can expose retrieve() as a tool so an AI assistant can search your documents on demand; our guide on how to build an MCP server in Python shows how to wrap a function like this for clients that support the Model Context Protocol.

What it means

RAG has become the standard way to make language models useful on private or fast-changing information, and the tooling is now simple enough to run on an ordinary computer. The roughly 100 lines above cover the core of many commercial "chat with your documents" products: chunking, embedding, vector search and a grounded prompt with citations.

The local approach changes the economics and the privacy picture. No per-query fees are charged, and documents stay on your machine. That matters for teams handling material they cannot send to an outside service, and for anyone weighing obligations under new rules such as the EU AI Act. The limit is model quality: a small local model will still make mistakes. That is why citations and a written evaluation checklist matter more than the choice of any one model. For a fuller workflow with graders and CI checks, see how to evaluate an LLM app in Python.

Sources

• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv / NeurIPS 2020)
• Ollama blog: Embedding models (8 April 2024)
• Ollama API reference: /api/embed and /api/chat
• Ollama library: nomic-embed-text
• Nomic AI: nomic-embed-text-v1.5 model card (Hugging Face)
• ollama-python library on GitHub
• Chroma documentation: Getting started
• Chroma documentation: Clients (PersistentClient)
• Chroma documentation: Configuring collections (distance functions)
• Sentence Transformers documentation