The Project | About Us | Contribute | Donations | License

HOME

This is the moment the course has been preparing for since 01-01. We have embeddings and semantic search (04-01), a persistent vector database with filters (04-02) and a well-chunked corpus with metadata (04-03). And from the earlier modules we have something just as valuable: a SYSTEM_DOCUBOT that since 02-01 has demanded that answers be based exclusively on the <documentation> block, an answer_prompt() that builds it, a JSON contract with a sources field, and a robust answer() with retries and graceful degradation (03-02). All of those pieces were designed with a hole in the middle: someone had to fill the <documentation> block — and until today that someone was us, by hand. In this lesson we close the circle: we build rag.py, the answer_with_rag() function that, given a question, automatically retrieves the relevant chunks, assembles the documentation block with its real sources and generates the answer. We will also make the design decisions that never appear in marketing diagrams (how many chunks? in what order? what if nothing is relevant?), integrate RAG with the conversations from 03-03, and settle an outstanding debt from 01-04: the PDF question that made the naive DocuBot hallucinate.

The full architecture: how the pieces fit
Retrieve: from the question to the chunks
Augment: building the <documentation> block with sources
Generate: rag.py and answer_with_rag()
Design decisions: k, order and the anti-noise threshold
RAG in conversations: integrating with Conversation
Before and after: the PDF question from 01-04

The full architecture: how the pieces fit

RAG (Retrieval-Augmented Generation) has three phases with names of their own: retrieve the relevant fragments, augment the prompt with them, generate the answer. Nothing in the third phase is new to us — it is exactly the DocuBot of modules 2 and 3. What is new is that the first two stop being manual:

flowchart TD
    P[User question] --> E["embed(question, kind='query')<br/>(embeddings.py, 04-01)"]
    E --> R["collection.query(top-k, filters)<br/>(docs_nubelia, 04-02/04-03)"]
    R --> U{"max similarity<br/>≥ threshold?"}
    U -- no --> F["Anti-hallucination response<br/>(rule 3 of SYSTEM_DOCUBOT)"]
    U -- yes --> D["Build &lt;documentation&gt; block<br/>with chunks + sources"]
    D --> PR["answer_prompt()<br/>(prompts.py, 02-01)"]
    PR --> G["answer(question, documentation)<br/>(docubot.py, 03-02: retries + degradation)"]
    G --> J["Validated JSON contract<br/>{category, answer, sources, confidence}<br/>with REAL sources from the metadata"]

It is worth pausing on what this diagram means for the course: we are not going to touch prompts.py, validation.py or docubot.py. The system prompt from 02-01 already talked about the <documentation> block back when we could only fill it by copy-paste; the answer() from 03-02 already took documentation as a parameter. The layered design is about to pay off: RAG is a new layer on top, not a rewrite.

Retrieve: from the question to the chunks

The first phase translates the question into the meaning space and queries the collection:

# rag.py (part 1) — retrieval
from embeddings import embed
from vectordb import get_collection

DEFAULT_K = 4              # how many chunks to retrieve (decision justified below)
RELEVANCE_THRESHOLD = 0.45   # minimum similarity of the best chunk (calibrate in 04-05)

def retrieve(question: str, k: int = DEFAULT_K, filters: dict | None = None) -> list[dict]:
    """Returns the k most relevant chunks, as a list of uniform dicts."""
    collection = get_collection()
    result = collection.query(
        query_embeddings=embed([question], kind="query"),
        n_results=k,
        where=filters,          # None = search the whole corpus
    )
    chunks = []
    for doc, meta, dist in zip(result["documents"][0],
                               result["metadatas"][0],
                               result["distances"][0]):
        chunks.append({
            "text": doc,
            "similarity": 1 - dist,    # cosine distance -> similarity (04-02)
            "url": meta["url"],
            "title": f"{meta['title']} > {meta['section']}",
            "version": meta.get("version", ""),
        })
    return chunks

Everything here is familiar material: embed with input_type="query" (04-01), the docs_nubelia collection (04-02), the distance→similarity conversion, and metadata coming from the ingestion (04-03). The function normalizes Chroma's "list of lists" format into a list of dicts the rest of the pipeline works with comfortably.

About the filters parameter: this is where the category filter using the 02-02 classifier would plug in, or the product-version filter. As we saw in exercise 2 of 04-02, a filter based on a wrong classification sabotages retrieval — which is why the default is None (search everything) and the filter is an option you enable with evidence, not an assumption you hardwire.

Augment: building the `<documentation>` block with sources

The second phase formats the chunks for the prompt. Each fragment goes in numbered, with its source visible, so the LLM can cite it:

# rag.py (part 2) — building the documentation block
def build_documentation(chunks: list[dict]) -> str:
    """Formats the retrieved chunks the way the <documentation> block expects."""
    parts = []
    for i, c in enumerate(chunks, start=1):
        parts.append(
            f"[Fragment {i} | source: {c['url']} | section: {c['title']}]\n{c['text']}"
        )
    return "\n\n---\n\n".join(parts)

Details with intent:

Numbering the fragments lets us instruct the model (via answer_prompt(), which already asks for cited sources) to refer to them unambiguously, and it eases diagnosis: when an answer is bad, we will know exactly what the model saw.
The source travels inside the block, but the sources field of the final JSON will not depend on the model copying it correctly: we will fill it ourselves from the metadata (next section). The model cites for the answer text; the system guarantees traceability.
The --- separator marks clear boundaries between fragments: cheap and effective against accidental content mixing.

Generate: `rag.py` and `answer_with_rag()`

The function that joins the three phases and that modules 5 and 6 will use as "the complete DocuBot":

# rag.py (part 3) — the complete pipeline
from prompts import answer_prompt, PROMPT_VERSION
from docubot import answer               # 03-02: retries, streaming, degradation
from validation import FALLBACK_RESPONSE

NO_INFO_PHRASE = "I can't find that information in the available documentation."

def answer_with_rag(question: str, k: int = DEFAULT_K,
                    filters: dict | None = None) -> dict:
    """Complete RAG pipeline: retrieve -> augment -> generate.

    Returns DocuBot's JSON contract:
    {"category", "answer", "sources", "confidence"}
    """
    # 1) RETRIEVE
    chunks = retrieve(question, k=k, filters=filters)

    # Guardrail: if not even the best chunk is relevant, we don't generate over noise
    if not chunks or chunks[0]["similarity"] < RELEVANCE_THRESHOLD:
        return {
            "category": "other",
            "answer": NO_INFO_PHRASE,
            "sources": [],
            "confidence": "low",
        }

    # 2) AUGMENT
    documentation = build_documentation(chunks)

    # 3) GENERATE — the robust answer() from 03-02, untouched
    response = answer(question, documentation)

    # 4) REAL sources: from the system, not from the model's memory
    if response != FALLBACK_RESPONSE:
        response["sources"] = sorted({c["url"] for c in chunks})

    return response

The four steps, annotated:

Step 1 and its guardrail. Retrieval always returns something — the k nearest neighbors exist even when the question is "Can I pay with cryptocurrency?". The right question is not "are there results?" but "are they relevant?". If the best chunk doesn't reach the threshold, we answer directly with the anti-hallucination phrase from rule 3 of SYSTEM_DOCUBOT — without spending an LLM call and without giving the model the chance to stretch an irrelevant chunk into a plausible, false answer. Note the order of the benefits: honesty first, savings second.
Step 3 inherits all the robustness of 03-02 for free: retries with backoff+jitter, degradation to FALLBACK_RESPONSE if everything fails, contract validation with clean_json/validate_contract. We don't duplicate a single line of that logic.
Step 4 is the reliability leap announced in 04-03: sources stops being what the model claims to have used and becomes what the system knows it gave it. A deduplicated, sorted set of URLs from the metadata. If a user doubts an answer tomorrow, those URLs are one click away from verification.
What about the cache? answer_with_rag is the natural place to consult SemanticCache (04-01) before step 1 and to save after step 4 — we leave it out of the listing for clarity, and as an exercise.

Design decisions: k, order and the anti-noise threshold

Three decisions every RAG pipeline makes, whether it makes them explicit or not. Better explicit:

Decision	Our choice	Reasoning	Risk of overdoing / underdoing it
How many chunks (k)?	`k=4`	With chunks of ~450 tokens (04-03), that is ~1,800 tokens of context: enough for questions touching 2 sections, affordable in cost (03-04)	High k: cost, dilution, "lost in the middle" (01-04). Low k: the answer may be in chunk #5
In what order?	Most relevant first	Models attend better to the start of the context; fragment 1 is the one almost certainly used	Random or reversed order gives away precision for nothing
What if nothing is relevant?	Threshold on the best chunk's similarity → anti-hallucination phrase	Retrieving noise and generating over it is manufacturing hallucinations "with sources"; better an honest no	High threshold: DocuBot becomes uselessly timid. Low threshold: hallucinations with citations return

About the value RELEVANCE_THRESHOLD = 0.45: it is a starting point, not a truth. As we learned in 04-01, the similarity scale depends on the embeddings model; the final value must come from measuring against the evaluation dataset from 02-04 — which, remember, has ~20% "no answer" cases precisely for this. In 04-05 we will turn that calibration into an experiment with numbers.

One honest mention to close the section: there is a technique called reranking — retrieving more candidates (e.g. 20) and reordering them with a dedicated model that compares question and chunk more finely than cosine does, keeping the best 4. It improves mature systems, adds one more component and one more latency hop, and for the size of Nubelia's corpus it is not the first lever to pull. We know it exists; we move on.

RAG in conversations: integrating with `Conversation`

DocuBot doesn't receive isolated questions: it holds conversations (the Conversation class and SessionManager, 03-03). How do RAG and history coexist? The short answer is the motto we coined in 03-03: "history for what was said, RAG for what is known". In practice:

We embed the turn's question, not the history. Embedding the whole conversation would contaminate the search with the topics of earlier turns.
But the raw question is sometimes not enough. After "How do I invite a member?", the user asks "and how do I remove them?" — an embedding of "and how do I remove them?" will not find the member-management docs. The solution is called query rewriting: a cheap preliminary LLM call that rephrases the question using the history's context ("How do I remove a member from a workspace?") before embedding it. We mention it so it has a name; a simple version is enough:

# rag.py (part 4) — per-turn RAG inside a conversation
def answer_in_conversation(conv, question: str, k: int = DEFAULT_K) -> dict:
    """One conversation turn with RAG: retrieve per turn, history kept separate."""
    search_question = rewrite_if_needed(conv, question)   # query rewriting
    chunks = retrieve(search_question, k=k)

    if not chunks or chunks[0]["similarity"] < RELEVANCE_THRESHOLD:
        response = {"category": "other", "answer": NO_INFO_PHRASE,
                    "sources": [], "confidence": "low"}
    else:
        documentation = build_documentation(chunks)
        # The documentation goes into the CURRENT turn; the history comes from conv
        response = answer(question, documentation, history=conv.messages())
        if response != FALLBACK_RESPONSE:
            response["sources"] = sorted({c["url"] for c in chunks})

    conv.add_turn(question, response)   # the 10-turn truncation (03-03) still applies
    return response

The important architectural point: the retrieved documentation belongs to the turn, not to the history. Each turn retrieves its own; the previous turn's chunks are not carried forward (they would bloat the context and the cost, and the truncation from 03-03 would sweep them away anyway). The history preserves what was said — questions and answers — and RAG supplies, on each turn, the knowledge that turn needs.

Before and after: the PDF question from 01-04

Let's settle the debt. In 01-04, the "naive DocuBot" — a bare call to the model, with no system prompt and no documentation — answered this to a question about reports:

Question: "How do I export a project report to PDF?" Naive DocuBot (01-04): "To export to PDF, go to Reports → Export → PDF and select the desired template. You can also use the API's /clone endpoint to duplicate the report before exporting it."

Two inventions in three lines: PDF export doesn't exist and neither does the /clone endpoint. Plausible, fluent, false. Now, the same case with the complete pipeline:

from rag import answer_with_rag
import json

print(json.dumps(answer_with_rag("How do I export a project report to PDF?"),
                 ensure_ascii=False, indent=2))

{
  "category": "other",
  "answer": "PDF export is not supported in Nubelia. Project reports can be exported to CSV and Excel from the Reports → Export menu (Fragment 1). If you need a PDF, one option is to export to Excel and print to PDF from your spreadsheet.",
  "sources": ["wiki/reports/export"],
  "confidence": "high"
}

It is worth dissecting why this works, piece by piece and lesson by lesson:

Retrieval (04-01/04-02) found the "Reports and export" chunk — which says explicitly that PDF is not supported, thanks to the ingestion (04-03) keeping the section whole.
Rule 2 of SYSTEM_DOCUBOT (02-01) forced the model to stick to that chunk: there is nowhere to get a nonexistent "Export → PDF" menu from.
The JSON contract (02-03) and its defensive validation returned a processable structure, and sources contains the chunk's real URL — put there by the system, not by the model.
And had the documentation said nothing about exporting, the threshold would have triggered the exact phrase from rule 3: "I can't find that information in the available documentation."

This before/after is the course's argument in miniature: the difference between an LLM that sounds like a Nubelia expert and a system that is one lies not in the model — it is the same one — but in everything we have built around it.

Common Mistakes and Tips

Always generating, whether or not retrieval was decent. It is the most expensive mistake in quality terms: the LLM will turn noise into convincing prose. The threshold on the best chunk's similarity is your net; calibrate it, but never remove it.
Letting the model invent the sources field. If sources comes out of generation, you will have pretty citations that are occasionally false. Fill it from the retrieved chunks' metadata: it is information the system holds with certainty.
Embedding the full history to retrieve in conversations. The search gets contaminated with the topics of every turn. Embed the turn's question (rewritten if it is elliptical).
Carrying previous turns' chunks forward in the context. Growing cost and a context bloated with documentation that is no longer on point. Documentation per turn; history for what was said.
Treating k and the threshold as universal constants copied from a tutorial. They are a function of your corpus, your chunking and your embeddings model. Lesson 04-05 exists to give them well-founded values.
Tip: log on every call (like the CSV from 02-04) the question, the ids and similarities of the retrieved chunks, and the answer. When a bad answer arrives — it will — you will be able to answer the key diagnostic question: did retrieval fail or did generation fail? That is exactly the distinction the next lesson is built on.

Exercises

Plug in the semantic cache. Modify answer_with_rag() to consult a SemanticCache instance (04-01) before retrieving, and to save the answer after generating (only if it is not FALLBACK_RESPONSE and its confidence is not "low" — reason through why those two conditions).
The threshold in action. With the demo corpus loaded, run answer_with_rag() with these three questions and note the best chunk's similarity and the behavior: (a) "How do I reset my password?", (b) "Can I pay for Nubelia with cryptocurrency?", (c) "What do you think of Real Madrid's latest signing?". Does the threshold tell the three cases apart? Which 02-04 case type does each represent?
Minimal query rewriting. Implement rewrite_if_needed(conv, question): if the conversation is empty, return the question as-is; otherwise, use ask_docubot() (03-01) with a short prompt asking to rewrite the question so it is self-contained, using the history. Try it with the dialogue "How do I invite a member?" → "and how do I remove them?".

Solutions

Skeleton of the integration:

from semantic_cache import SemanticCache
_cache = SemanticCache(threshold=0.92, ttl_seconds=3600)

def answer_with_rag(question, k=DEFAULT_K, filters=None):
    cached = _cache.get(question)
    if cached is not None:
        return cached
    # ... pipeline as before ...
    if response != FALLBACK_RESPONSE and response.get("confidence") != "low":
        _cache.save(question, response)
    return response

Why the two conditions: caching the fallback would freeze a transient error (an API error spike from 03-02) into the official answer for the whole TTL; and caching low-confidence answers multiplies the reach of the system's most dubious responses — a cache should amplify successes, not doubts. Note the order: the cache is consulted before retrieving, so a hit saves the retrieval embedding, the Chroma query and the LLM call.

Typical results: (a) similarity ~0.75-0.85 → well above the threshold, normal answer with source wiki/administration/security-and-access — an easy case from 02-04. (b) similarity ~0.35-0.45 → borderline zone: there are billing/plans chunks that are somewhat similar; depending on which side of the threshold it lands, DocuBot answers with the anti-hallucination phrase or tries to answer with the plans docs (and if the prompt does its job, it will say there is no information about cryptocurrency) — a no-answer case, and the best demonstration of why the threshold is calibrated with data rather than by eye. (c) similarity ~0.10-0.20 → clearly below the threshold, anti-hallucination phrase without spending a call — an out-of-domain case (a relative of the soft adversarial ones). If your threshold doesn't separate (a) from (c) with room to spare, look at the chunking before the threshold.
A minimal, sufficient version:

def rewrite_if_needed(conv, question: str) -> str:
    if not conv.messages():
        return question
    instruction = (
        "Rewrite the user's last question as a self-contained question, "
        "resolving pronouns and references with the help of the history. "
        "Return ONLY the rewritten question.\n\n"
        f"History:\n{conv.summary_text()}\n\nLast question: {question}"
    )
    return ask_docubot(instruction).strip()

With the dialogue from the exercise, the expected rewrite is "How do I remove a member from a workspace in Nubelia?" — and that one does retrieve the right docs. Cost: one extra short call per conversational turn; enable it only when there is history (the if condition) so you don't pay for it on standalone questions. Query rewriting goes much further (query expansion, multi-question), but this version covers the case that genuinely breaks conversational RAG: elliptical questions.

Conclusion

DocuBot is now what we promised in 01-01: an assistant that genuinely knows Nubelia. rag.py closes the circle with answer_with_rag() — retrieve (embedding of the question + top-k over docs_nubelia), augment (a numbered <documentation> block with sources) and generate (the robust answer() from 03-02, untouched) — returning the same JSON contract as always but with a difference of a whole different order: the sources are now facts from the system, not claims from the model. Along the way we made the decisions that define a serious RAG (a reasoned k=4, relevance-first ordering, and the threshold that prefers an honest "I can't find that information in the available documentation." over generating on noise), we integrated RAG with the 03-03 conversations under the motto "history for what was said, RAG for what is known" (with query rewriting for elliptical questions), and we settled the debt from 01-04: the PDF question that used to produce invented menus and endpoints now produces a correct answer, with a verifiable source. But let's be as honest as we were with the naive DocuBot: the pipeline working in our demos doesn't mean it works. Does it retrieve the right chunks 95% of the time, or 70%? Is the 0.45 threshold wise, or a coincidence? Are the answers faithful to the chunks, or does the model still embellish? A RAG system has two engines that can fail independently — retrieval and generation — and evaluating them requires separating them. That is precisely the job of the module's final lesson: evaluating RAG systems (04-05).

Building a complete RAG pipeline

Contents

The full architecture: how the pieces fit

Retrieve: from the question to the chunks

Augment: building the `<documentation>` block with sources

Generate: `rag.py` and `answer_with_rag()`

Design decisions: k, order and the anti-noise threshold

RAG in conversations: integrating with `Conversation`

Before and after: the PDF question from 01-04

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Generative AI and LLMs for Developers Course

Module 1: Generative AI Fundamentals

Module 2: Prompt Engineering

Module 3: Integrating LLMs into Applications

Module 4: RAG - Retrieval-Augmented Generation

Module 5: Function Calling and Agents

Module 6: Toward Production

Building a complete RAG pipeline

Contents

The full architecture: how the pieces fit

Retrieve: from the question to the chunks

Augment: building the <documentation> block with sources

Generate: rag.py and answer_with_rag()

Design decisions: k, order and the anti-noise threshold

RAG in conversations: integrating with Conversation

Before and after: the PDF question from 01-04

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Generative AI and LLMs for Developers Course

Module 1: Generative AI Fundamentals

Module 2: Prompt Engineering

Module 3: Integrating LLMs into Applications

Module 4: RAG - Retrieval-Augmented Generation

Module 5: Function Calling and Agents

Module 6: Toward Production

Augment: building the `<documentation>` block with sources

Generate: `rag.py` and `answer_with_rag()`

RAG in conversations: integrating with `Conversation`