In 04-01 we left a brute-force semantic search working, with two confessed embarrassments: every query compares the question against all the vectors in the corpus, and those vectors live in a Python list that evaporates when the process restarts, forcing us to re-embed (and re-pay for) the whole documentation set on every startup. For the six-fragment mini-corpus it didn't matter; for the hundreds of pages in Nubelia's wiki, it does. This lesson introduces the piece of infrastructure that solves both problems: the vector database. We will see exactly what it contributes (persistence, approximate indexes, metadata with filtering), set up ChromaDB locally step by step — it will be DocuBot's vector database for the rest of the course — understand at a conceptual level the accuracy/speed trade-off of ANN indexes, and close with a map of alternatives so you can choose with good judgment when the project isn't DocuBot.

1. Why brute force doesn't scale
2. What a vector database contributes
3. ANN indexes: trading a pinch of accuracy for a lot of speed
4. ChromaDB step by step: DocuBot's collection
5. Metadata and filtering: searching only where it makes sense
6. The landscape of alternatives: how to choose

Let's put numbers to the problem. Our search() function from 04-01 performs, for every query, one cosine similarity against every vector in the corpus — cost O(n): it grows linearly with the number of fragments.

And latency isn't the only problem. With the in-memory list:

• Every startup re-embeds the whole corpus: minutes of waiting and an embeddings bill paid over and over for no reason — exactly the kind of avoidable expense we learned to hunt down in 03-04.
• No incremental updates: if Nubelia publishes a new page, either you re-embed everything or you invent manual bookkeeping of what is embedded and what isn't.
• No filters: if the user asks about product version 2026.2, brute force may cheerfully return documentation from 2024, because all it looks at is closeness of meaning.
• A single process: two DocuBot instances cannot share the list.

This should sound familiar: these are the same reasons applications don't keep their data in Python dictionaries but in a database. Vectors are no exception — they just need a database that knows how to perform the operation "give me the k nearest neighbors" efficiently.

A vector database is a store specialized in vectors + their text + their metadata, with three superpowers:

1. Persistence. The vectors are saved to disk. Embedding the corpus is paid for once (during ingestion, which we will build in 04-03) and queries reuse it forever. Restarting the service costs zero.
2. ANN indexes (Approximate Nearest Neighbors). Instead of comparing against everything, the index organizes the vectors so it can find the closest ones while visiting only a small fraction — search goes from O(n) to something close to logarithmic. The word "approximate" comes with fine print; we unpack it in the next section.
3. Metadata and filtering. Each vector carries an attached metadata dictionary (category, version, source URL, date...) and queries can combine semantic closeness with exact filters: "the 4 chunks most similar to this question that belong to the permissions category and version 2026.2". For DocuBot this connects directly with the 6-category classifier from 02-02: classifying the question first lets us search only the relevant section of the documentation.

On top of that, depending on the product: an upsert API (insert-or-update, key for re-ingestions), deletion by id, replicas, access control... The essential thing to understand is that the vector database does not compute embeddings (that remains the job of our embeddings.py, although some offer to do it for convenience) nor generate answers (that's the LLM): it is the middle piece that remembers and finds.

How do you find the nearest neighbor without looking at all the vectors? The general idea behind ANN indexes is to pre-organize the vectors into a structure that lets you discard entire regions of the space without visiting them.

The most popular index today is HNSW (Hierarchical Navigable Small World), used by Chroma, Qdrant, Weaviate and pgvector. Without going into the math, the intuition:

• Picture the vectors as cities on the meaning map, connected by roads to their nearby neighbors.
• HNSW adds layers of highways on top: an upper layer with a few widely spaced cities and long-distance connections, denser intermediate layers, and the full local road network at the bottom.
• A search enters via the highway (long jumps toward the right region), descends layer by layer (ever finer jumps) and ends up exploring only the local neighborhood of the answer.
• Result: instead of visiting all 50,000 vectors, it visits a few hundred.

The fine print is the word approximate: the index does not guarantee finding exactly the k nearest neighbors; it guarantees finding almost all of them almost always. The metric that measures this is called the index's recall (percentage of the true k neighbors the index returns — don't confuse it yet with retrieval recall from 04-05, which measures something else at another level).

Two practical consequences, so you don't obsess:

• At DocuBot's scale (thousands of chunks), the defaults of any vector database give extremely high recall with millisecond latency. Don't touch the index parameters until you have a measurable problem.
• Occasionally missing the 4th nearest neighbor almost never matters in RAG: if your answer depends on retrieving exactly that chunk and no similar one, the real problem usually lies in the chunking or in corpus redundancy, not in the index.

ChromaDB is an open source vector database that runs locally (embedded in your Python process or as a server), persists to disk and has a minimal API — ideal for development and for projects of DocuBot's size. Installation:

We create the module that will centralize access (modules 4, 5 and 6 will import it as-is):

Breakdown:

• PersistentClient(path=...) stores everything in the given directory. That is the persistence: kill the process, start it again, and the vectors are still there. (There is also chromadb.Client(), purely in-memory, useful only for tests.)
• get_or_create_collection: a collection is the equivalent of a table — a set of vectors backed by the same embeddings model. docs_nubelia will be our canonical collection.
• {"hnsw:space": "cosine"}: we tell the HNSW index to measure with cosine, consistent with 04-01. Watch out for a nuance that causes confusion: Chroma returns distances, not similarities, and with cosine space the relationship is similarity = 1 - distance. Distance 0.2 ⇒ similarity 0.8.

Now we add documents. A deliberate design point: we compute the embeddings ourselves, with the embed() from 04-01, and hand them to Chroma ready-made. Chroma can compute them on its own with a default local model, but we would lose control over which model is used — and we already know from 04-01 that mixing embeddings models is a silent, lethal mistake.

Three important details:

• upsert instead of add: if the id already exists, it updates instead of failing or duplicating. This is what will make the ingestion in 04-03 re-runnable.
• The ids are our responsibility, and they should be deterministic (derived from the chunk's origin), not random: that way the same page, re-ingested, overwrites its own chunks.
• Chroma stores the text alongside the vector (documents): when querying, we get the fragment back directly for the <documentation> block, without a second database.

And the query:

Note the shape of the result: each field is a list of lists (result["documents"][0]), because query accepts several queries at once and returns one list of results per query. Since we sent a single question, we always index with [0]. This is the number one stumbling block with Chroma's API.

The where parameter combines semantic search with exact filters over the metadata:

When should DocuBot filter? Two cases with real potential:

• By category, reusing the classifier from 02-02: if classification_prompt() says the question is about billing, searching only that category eliminates at the root the risk of an API chunk "sneaking in" through superficial similarity. (In 04-04 we will decide when it pays to enable this filter and when it is better to search everything.)
• By product version: when documentation for the current version coexists with older versions, the filter prevents answering with obsolete instructions — an especially damaging kind of error in support.

A symmetric warning: every filter is a bet. If the classifier picks the wrong category, the filter hides precisely the right chunks and retrieval fails silently. Filter when the metadata is reliable (the version is; an automatic classification, only sometimes), and measure the effect with the evaluation in 04-05 before locking it in.

Chroma is our didactic choice and perfectly valid for DocuBot, but the ecosystem is broad. The honest thing is not a ranking but a map of criteria:

The decision criteria, in a reasonable order of importance:

1. Local or managed? The same dilemma as in 01-03 with LLMs: control and data privacy versus operational convenience. For internal documentation, where the vectors live matters (embeddings are not encryption: much of the content can be reconstructed from a vector).
2. What does your team already operate? If there is PostgreSQL in production, pgvector often wins on operational simplicity even if another option looks "better" on paper.
3. Real scale (not aspirational): thousands of vectors → anything works, start simple; tens of millions → dedicated server or managed service.
4. Filtering and features: if your queries depend on rich filters or hybrid search, look at Qdrant/Weaviate first.

The good news: since our code isolates access in vectordb.py and embeddings in embeddings.py, migrating to another vector database later is a localized change, not a rewrite.

• Confusing distance with similarity. Chroma returns distances; with cosine space, similarity = 1 - distance. If your "similarities" look better the lower they are, you are reading distances. Convert in a single place and always work in similarity.
• Forgetting metadata={"hnsw:space": "cosine"} when creating the collection: the default space may not be cosine, and the numbers will stop matching those from 04-01. It is fixed at collection creation — changing it later means recreating the collection.
• Letting the database compute embeddings with its default model. You will end up with a corpus embedded with one model and questions embedded with another: bad results with no visible error. Always pass embeddings= computed with your own embed().
• Random ids (uuid4) at load time. Every re-ingestion will duplicate the entire corpus. Deterministic ids + upsert is the right pairing.
• Forgetting the [0] when reading query results (they are lists of lists, one per query).
• Optimizing the ANN index prematurely. At DocuBot's scale, the defaults are more than enough. Measure first (04-05), tune later.
• Tip: store in the metadata everything you might need for filtering or citing (category, version, URL, date). Adding a metadata field after the fact means re-processing the entire corpus.

1. Persistence demonstrated. Run populate_demo.py, close the Python interpreter, open it again and, without adding anything, run a query against get_collection(). Check with collection.count() that the 4 documents are still there. What would have happened with the VECTORS list from 04-01?
2. The filter that helps and the filter that hurts. With the 4 demo documents, query "How many tasks can I have in a project?" first without a filter and then with where={"category": "api"}. Compare the results and explain what practical lesson this teaches about filters based on automatic classification.
3. Idempotent upsert. Run populate_demo.py three times in a row and check collection.count(). Then change the text of document reports-export-01 (for example, add "PDF export is planned for 2027."), re-run, and query "Can I export to PDF?" to verify you retrieve the new version. Which two decisions in the code make this possible?

1. count() returns 4 and the query works: PersistentClient wrote the vectors to ./docubot_db and reloads them from disk. The VECTORS list from 04-01 would have vanished with the process, and rebuilding it would have required calling the embeddings API again for the whole corpus — paying in time and money. That is exactly the difference between an in-memory structure and a database.

2. Without a filter, the "The Pro plan allows up to 500 tasks per project" chunk comes first with clearly higher similarity — correct retrieval. With where={"category": "api"}, that chunk is excluded at the root (its category is billing) and the query returns the API authentication chunk, which is the most similar thing within the filter but irrelevant to the question. Lesson: a filter based on a wrong classification doesn't degrade the search — it sabotages it, and moreover with low similarities that look like "normal results". That is why in 04-04 we will treat the category filter as optional and conditioned on the classifier's confidence, and why the minimum similarity threshold from exercise 1 of 04-01 will be our safety net.

3. count() is still 4 after three runs: upsert with the same ids overwrites instead of duplicating (had we used add with new ids each time, we would have 12). When the text of reports-export-01 changes, the re-run recomputes its embedding and overwrites vector, text and metadata under the same id, so the query retrieves the updated version. The two decisions: deterministic ids (the document "knows" what its id is) and upsert. They are the foundation of the re-runnable ingestion script in the next lesson — together with a third step we performed only mentally here: invalidating the response caches from 03-04/04-01, because the documentation just changed.

DocuBot now has long-term memory: ChromaDB's docs_nubelia collection persists the vectors on disk (embed once, query forever), finds them in milliseconds thanks to an HNSW index that trades a pinch of accuracy for orders of magnitude of speed, and stores alongside each vector the original text plus metadata (category, version, URL) that make search filterable and will soon feed the sources field of the JSON contract. We encapsulated access in vectordb.py::get_collection(), learned to convert Chroma's distances into 04-01's similarities, and placed Chroma on a map of alternatives (pgvector, Qdrant, Weaviate, Pinecone, FAISS) where the right choice depends on criteria — local or managed data, what the team already operates, real scale — and not on rankings. But our collection is still populated with four handwritten fragments in a demo script. Nubelia's real documentation is hundreds of markdown pages across the wiki and the repos, with headings, tables and sections of wildly varying lengths. Turning that raw material into well-cut chunks, with rich metadata, through a re-runnable ingestion script is the craft we learn in the next lesson: document ingestion and chunking (04-03).