The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Document ingestion and chunking

Next ►
►

We have a vector database (docs_nubelia, 04-02) and we know how to embed and compare meanings (04-01), but the collection is still populated with four handwritten toy fragments. Nubelia's real documentation is a different beast: hundreds of markdown pages spread across the wiki and the repos, with step-by-step guides, API reference, release notes, nested headings and sections ranging from two lines to two thousand words. This lesson covers the craft that turns that raw material into a queryable corpus: why documents must be split (it is not optional), how to split them well (the chunking strategies and their trade-offs), which metadata each piece must carry (spoiler: it is what will fill the sources field of the JSON contract), and how to package it all into a re-runnable ingestion script — because the documentation changes every week and the corpus has to keep up without drama. The quality of the entire RAG pipeline we will assemble in 04-04 is decided, in large part, right here.

Contents

  1. The Nubelia corpus: what we are dealing with
  2. Why split: context, precision and cost
  3. Chunking strategies: a comparison
  4. Implementation: heading-based chunking with overlap
  5. Per-chunk metadata: the origin of the sources
  6. The ingestion pipeline: ingestion.py
  7. Re-ingestions: upsert and cache invalidation

The Nubelia corpus: what we are dealing with

Before writing code, an inventory. Nubelia's documentation — scattered, as we confessed in 01-01 — has been consolidated into a directory of markdown files exported from the wiki and the repos:

docs_nubelia/
├── guides/
│   ├── getting-started.md
│   ├── task-management.md
│   └── reports-and-export.md
├── administration/
│   ├── security-and-access.md
│   ├── members-and-permissions.md
│   └── plans-and-billing.md
├── api/
│   ├── authentication.md
│   ├── project-endpoints.md
│   └── webhooks.md
└── integrations/
    ├── slack.md
    └── calendar.md

Each file follows a reasonable convention (not a perfect one — this is real documentation): one # Title per page, ## sections and ### subsections, and a first comment line with publication metadata that the wiki adds on export:

<!-- url: wiki/administration/security-and-access | version: 2026.2 | updated: 2026-06-15 -->
# Security and access

## Reset password

To reset your password, go to **Settings → Security**...

## Account lockout

If your account gets locked after several failed attempts...

We work with markdown because it is the most common case (wikis, READMEs, docs portals) and because its heading structure is pure gold for chunking. If your real corpus includes PDF, HTML or Word, the pattern is identical with a prior conversion step to text/markdown (libraries such as pypandoc, beautifulsoup4 or PDF extractors); we won't develop that here so as not to lose focus.

Why split: context, precision and cost

The initial temptation is not to split at all: embed each full page as a single vector. Three reasons rule that out:

  1. Context limits. A whole page may not fit in the embeddings model's window (04-01), and even if it does, several whole pages concatenated in the <documentation> block can overflow — or degrade — the LLM's window: remember "lost in the middle" from 01-04. We want to give the LLM little and good, not much and diluted.
  2. Retrieval precision. The embedding of a page that talks about passwords, lockouts, sessions and auditing is an average of all those topics: it isn't particularly close to any specific question. The embedding of the "Reset password" section points like a laser at password questions. Topically focused chunks ⇒ sharper similarities ⇒ better top-k.
  3. Cost. Every token in the <documentation> block is paid for on every call (we measured it with estimate_cost in 03-04). Retrieving 4 chunks of ~300 tokens instead of 4 pages of ~3,000 divides that part of the bill by ten — and it is the part that repeats with every question.

Splitting has a cost of its own too: a bad cut can separate an instruction from its condition ("...click Reset." / [cut] / "Note: this option requires administrator privileges."). Everything in this chapter's design aims to minimize that damage.

Chunking strategies: a comparison

The three families that cover virtually every case in practice:

Strategy How it cuts Advantages Drawbacks When to use it
Fixed size with overlap Every N tokens/characters, repeating the last M in the next chunk Trivial to implement; chunks of uniform, predictable size Cuts blindly: splits sentences, mixes topics; overlap mitigates but doesn't cure Unstructured text (transcripts, dumps) or as a quick first version
By structure (markdown headings) At ##/### section boundaries Respects the topical boundaries the author marked; chunks that make sense on their own; the section title is free metadata Sections of wildly uneven size: some huge (need sub-splitting), others tiny Structured documentation — our case, and the most common one in corporate corpora
By paragraphs / semantic At paragraph breaks; the "semantic" variant cuts where the embedding between consecutive sentences changes sharply Fine granularity; the semantic variant detects topic shifts without depending on formatting Chunks sometimes too small (lose context); the semantic variant embeds sentence by sentence: more cost and complexity Long texts without headings; a refinement when structure isn't enough

Two parameters that cut across every strategy:

  • Target size. The usual compromise sits between 200 and 600 tokens per chunk: enough for the fragment to stand on its own, small enough to point at one topic. There is no magic number — in 04-05 we will treat it as an experimental variable.
  • Overlap. Repeating the end of one chunk at the start of the next (10–20% of the size) ensures that a sentence split by a cut exists whole in at least one of the two chunks. The price: some duplicated storage and the risk of retrieving two nearly identical chunks in the same top-k.

For Nubelia we choose the strategy that exploits what we already have: by headings, with overlap-based sub-splitting for long sections. The best of two families.

Implementation: heading-based chunking with overlap

The plan for the function: split the document by ##/### headings, and if a section exceeds the maximum size, sub-split it by paragraphs with overlap. Each chunk comes out with its text and its metadata.

# chunking.py — heading-based markdown chunking with overlap
import re

MAX_SIZE = 1800    # maximum chunk size in characters (~450 tokens)
OVERLAP = 300      # characters repeated between consecutive sub-chunks

def chunk_markdown(text: str, url: str, version: str, date: str) -> list[dict]:
    """Converts a markdown document into a list of chunks with metadata."""
    page_title, sections = _split_by_headings(text)
    chunks = []
    for section_title, body in sections:
        for index, piece in enumerate(_sub_chunk(body)):
            chunks.append({
                # Explicit context: the chunk "knows" which page and section it comes from
                "text": f"[{page_title} > {section_title}]\n{piece}",
                "metadata": {
                    "section": section_title,
                    "title": page_title,
                    "url": url,
                    "version": version,
                    "date": date,
                },
                # Deterministic id: same section => same id on every re-ingestion
                "id": f"{url.replace('/', '-')}--{_slug(section_title)}--{index:02d}",
            })
    return chunks

def _split_by_headings(text: str):
    """Splits the markdown into (page_title, [(section_title, body), ...])."""
    page_title = "Untitled"
    m = re.search(r"^# (.+)$", text, flags=re.MULTILINE)
    if m:
        page_title = m.group(1).strip()
    # Split on level 2 or 3 headings, keeping each section's title
    parts = re.split(r"^(#{2,3} .+)$", text, flags=re.MULTILINE)
    sections, current_title = [], "Introduction"   # text before the first ##
    for part in parts:
        if re.match(r"^#{2,3} ", part):
            current_title = part.lstrip("#").strip()
        elif part.strip():
            sections.append((current_title, part.strip()))
    return page_title, sections

def _sub_chunk(body: str) -> list[str]:
    """If the section fits, a single chunk; otherwise, cut by paragraphs with overlap."""
    if len(body) <= MAX_SIZE:
        return [body]
    pieces, current = [], ""
    for paragraph in body.split("\n\n"):           # we never split inside a paragraph
        if len(current) + len(paragraph) > MAX_SIZE and current:
            pieces.append(current.strip())
            current = current[-OVERLAP:]           # the overlap: carry over the previous tail
        current += "\n\n" + paragraph
    if current.strip():
        pieces.append(current.strip())
    return pieces

def _slug(text: str) -> str:
    return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")

Decisions worth understanding (they are what separates correct chunking from mediocre chunking):

  • The [Page > Section] prefix inside the chunk text. A chunk that starts with "To reset your password..." is good; one that starts with "[Security and access > Reset password]\nTo reset..." is better: the context enters the embedding (improving similarity with questions that mention the general topic) and the LLM will see it while writing, reducing ambiguity. A cheap technique with a high return.
  • We never split inside a paragraph. The sub-splitting accumulates whole paragraphs; the worst case is a chunk slightly longer than MAX_SIZE, not a decapitated sentence.
  • The overlap carries over between sub-chunks of the same section (current[-OVERLAP:]), not across different sections: crossing a heading boundary would mix topics, exactly what the strategy avoids.
  • Deterministic ids (url--section--index): the piece we identified in 04-02 as a requirement for idempotent upsert.

Per-chunk metadata: the origin of the sources

This deserves its own section because metadata is not decoration — each field has a concrete consumer, already built or already announced:

Metadata field Who consumes it What for
url The sources field of the JSON contract (02-03) So DocuBot's answer cites the real page it came from — verifiable by the user
section, title The <documentation> block (04-04) and sources Readable citations ("Security and access > Reset password") instead of raw URLs
version The where filters from 04-02 Not answering with documentation for a version the user doesn't have
date Filters and diagnostics Detecting stale documentation; breaking ties between duplicate chunks

Until now, the sources field of the JSON contract was whatever the LLM claimed to have used — with documentation pasted in by hand, barely verifiable. From 04-04 onward, sources will be filled from this metadata, that is, from facts: which chunks were actually retrieved. It is a leap in reliability that gets decided here, in the ingestion, by properly recording where each piece came from.

The ingestion pipeline: ingestion.py

We assemble everything into the script that will run every time the documentation changes:

flowchart LR
    A[Read .md files] --> B[Clean and extract metadata]
    B --> C[Split with chunking.py]
    C --> D[Embed in batches with embeddings.py]
    D --> E[Upsert into docs_nubelia]
    E --> F[Invalidate response caches]
# ingestion.py — re-runnable ingestion pipeline for Nubelia's documentation
import os
import re
import pathlib
from chunking import chunk_markdown
from embeddings import embed
from vectordb import get_collection

DOCS_DIR = os.environ.get("DOCUBOT_DOCS", "./docs_nubelia")
BATCH_SIZE = 64   # how many chunks we embed per API call

def clean(text: str) -> str:
    """Normalizes the markdown before splitting."""
    text = re.sub(r"<!--.*?-->", "", text, flags=re.DOTALL)  # HTML comments out
    text = re.sub(r"\n{3,}", "\n\n", text)                   # collapse blank lines
    return text.strip()

def extract_publication_metadata(text: str, path: pathlib.Path) -> dict:
    """Reads the wiki's '<!-- url: ... | version: ... | updated: ... -->' line."""
    m = re.search(r"<!--\s*url:\s*(\S+)\s*\|\s*version:\s*(\S+)\s*\|\s*updated:\s*(\S+)\s*-->", text)
    if m:
        return {"url": m.group(1), "version": m.group(2), "date": m.group(3)}
    # Fallback values if the page lacks the line (real documentation = imperfect)
    return {"url": str(path.with_suffix("")).replace("\\", "/"), "version": "unknown", "date": "unknown"}

def ingest():
    collection = get_collection()
    all_chunks = []
    for path in sorted(pathlib.Path(DOCS_DIR).rglob("*.md")):
        raw = path.read_text(encoding="utf-8")
        meta = extract_publication_metadata(raw, path.relative_to(DOCS_DIR))
        chunks = chunk_markdown(clean(raw), **meta)
        all_chunks.extend(chunks)
        print(f"  {path.name}: {len(chunks)} chunks")

    # Embed and store IN BATCHES: fewer calls, less latency, same cost per token
    for i in range(0, len(all_chunks), BATCH_SIZE):
        batch = all_chunks[i : i + BATCH_SIZE]
        vectors = embed([c["text"] for c in batch], kind="document")
        collection.upsert(
            ids=[c["id"] for c in batch],
            documents=[c["text"] for c in batch],
            embeddings=vectors,
            metadatas=[c["metadata"] for c in batch],
        )
    print(f"Ingestion complete: {len(all_chunks)} chunks into '{collection.name}' ({collection.count()} total).")

    # The documentation has changed => cached responses may be stale
    invalidate_caches()

def invalidate_caches():
    """Rule from 03-04: publishing documentation invalidates cached responses."""
    from semantic_cache import SemanticCache   # plus the exact ResponseCache from 03-04
    # In a real deployment, this is where the shared store of both caches would be flushed.
    print("Response caches invalidated (ResponseCache + SemanticCache).")

if __name__ == "__main__":
    ingest()

A guided reading of the script:

  • clean() is minimal on purpose: it removes HTML comments (their metadata already extracted) and collapses whitespace. Aggressive cleaning (stripping tables, code, emojis) tends to destroy useful information; clean only what proves to be in the way.
  • Batches of 64: the pattern from 04-01 in action. With ~2,000 chunks that is ~32 calls to the embeddings API instead of 2,000. The cost per token is the same; the total latency and reliability, incomparable. (If the embeddings API returns transient errors, wrap embed with call_with_retries from retries.py — the patterns from 03-02 work for any API, not just the LLM's.)
  • Batched upsert with deterministic ids: running ingestion.py twice leaves the collection exactly the same; running it after editing a page overwrites only that page's chunks.
  • The per-file print is not cosmetic: "task-management.md: 47 chunks" when everything else has 8-12 is the signal of a monster page that may deserve restructuring — ingestion also doubles as a documentation-quality informant.

Re-ingestions: upsert and cache invalidation

Nubelia's documentation changes every week; the corpus must be updatable with a single command and no surprise side effects. Our design guarantees this with three properties:

  1. Idempotence: re-running with no changes duplicates nothing (deterministic ids + upsert, from 04-02).
  2. Real updates: an edited page regenerates its chunks under the same ids, with fresh embeddings and metadata.
  3. Consistency with the caches: the rule we fixed in 03-04 — publishing documentation invalidates ResponseCache — now extends to SemanticCache (04-01), and ingestion.py is the natural place to trigger it: it is the only point in the system that knows the documentation just changed. A cache that survives a re-ingestion can serve answers based on documentation that no longer exists — the worst possible bug in a support assistant, because nobody sees it.

One honest rough edge remains: if a section is deleted from a page (or a page disappears), its old chunks stay in the collection with orphaned ids — upsert updates and adds, but does not delete what is no longer generated. The robust solution is to compare the ids generated during ingestion with the existing ones (collection.get()) and delete the difference; we leave it flagged as an exercise, because it is exactly the kind of detail that separates a demo script from an operable piece.

Common Mistakes and Tips

  • Chunks with no context of their own. A chunk that says "Then click OK and the change is applied" is retrieved by no question and helps no LLM. The [Page > Section] prefix and the heading-based cut exist to prevent this; eyeball a sample of chunks after every strategy change.
  • Splitting by fixed size when structure is available. If the document has headings, use them: cutting a well-structured markdown "every 1,000 characters" throws away design information the author already gave you.
  • Huge chunks "so nothing is lost". You will retrieve fragments that contain the answer... diluted among three other topics, with flat similarities and a bloated (and paid-for, 03-04) <documentation> block. If torn between big and small, start small-to-medium and let the evaluation in 04-05 decide.
  • Non-deterministic ids, or ids derived from a global counter. A counter (chunk-0001, chunk-0002...) shifts its assignments as soon as a page is added early in the traversal: the upserts will overwrite the wrong chunks. The id must derive from stable content (url + section + index).
  • Forgetting cache invalidation on re-ingestion. The symptom is diabolical: the collection has the new docs, direct tests against the pipeline work, but users keep receiving old (cached) answers. Invalidating inside ingestion.py closes that door.
  • Tip: version chunking.py carefully and record (as with PROMPT_VERSION in 02-04) which chunking version generated the current corpus. Changing the chunking strategy changes the ids and the chunk contents: it demands a full re-ingestion and re-evaluation.

Exercises

  1. Chunk audit. Run chunk_markdown() on the example page security-and-access.md and print, for each chunk: id, first 80 characters and length. Verify that (a) no chunk exceeds MAX_SIZE by more than one paragraph, (b) the ids are unique, and (c) the [Page > Section] prefix is present. Then duplicate the content of one section until it exceeds MAX_SIZE and check that sub-splitting with overlap kicks in.
  2. Orphan deletion. Implement purge_orphans(collection, current_ids), which deletes from the collection the chunks whose id is not in the current ingestion, and call it at the end of ingest(). Hint: collection.get() with no filters returns the existing ids, and collection.delete(ids=[...]) deletes by id.
  3. Chunking as a variable. Without running anything: for the question "How long is the password reset link valid?", whose answer ("24 hours") sits in the last sentence of a long section, reason through what would happen with (a) whole-page chunks, (b) fixed 300-character size with no overlap, (c) our strategy. Which retrieves best and why?

Solutions

  1. With the example page, the expected output looks like:
wiki-administration-security-and-access--reset-password--00   | [Security and access > Reset password]\nTo reset your pas... | 412
wiki-administration-security-and-access--account-lockout--00  | [Security and access > Account lockout]\nIf your account... | 388

When you inflate a section past MAX_SIZE, ids --00, --01... appear for the same section, and the start of chunk --01 repeats the last ~300 characters of --00 (the overlap). If you also notice that chunk --01 starts mid-idea, you have just seen live why the overlap exists: the complete idea is guaranteed to be in --00.

  1. A straightforward implementation:
def purge_orphans(collection, current_ids: set[str]):
    existing = set(collection.get()["ids"])
    orphans = list(existing - current_ids)
    if orphans:
        collection.delete(ids=orphans)
        print(f"Purged {len(orphans)} orphan chunks.")

And in ingest(), after the upsert loop: purge_orphans(collection, {c['id'] for c in all_chunks}). With this, ingestion goes from "add and update" to a faithful mirror of the documentation directory. An operational nuance: if the ingestion runs against an accidentally incomplete directory (e.g. a partial checkout), the purge would delete half the corpus — a reasonable guardrail is to abort if current_ids is suspiciously small compared to the existing ones. Destructive decisions deserve belt and suspenders, like the budget ceiling from 03-04.

  1. (a) Whole page: the page's embedding averages passwords, lockouts and auditing; similarity with the question will be mediocre and, even if retrieved, the LLM receives 3,000 tokens with the key fact buried at the end — "lost in the middle" risk (01-04) and maximum cost. (b) Fixed 300 with no overlap: high probability that the cut falls between "you'll receive an email with a link" and "valid for 24 hours" — the retrieved chunk contains half the answer and DocuBot, faithful to rule 2 of SYSTEM_DOCUBOT, will be unable to answer precisely. (c) Our strategy: the complete "Reset password" section is one chunk (or two with overlap), topically pure, with the 24-hours sentence intact in at least one chunk, and with the section prefix reinforcing the similarity. It retrieves best because every decision (cutting at topical boundaries, never splitting paragraphs, overlapping) protects exactly the case the other two break.

Conclusion

The docs_nubelia collection no longer holds four demo sentences but Nubelia's real documentation, turned into chunks with craft: split by headings (respecting the topical boundaries the author marked), sub-split with overlap when a section runs long, prefixed with their [Page > Section] so the context travels inside the embedding, and loaded with the metadata — url, section, version, date — that will feed the filters from 04-02 and, above all, the sources field of the JSON contract with facts instead of claims. All orchestrated by ingestion.py: read → clean → split → embed in batches → upsert, re-runnable at will thanks to deterministic ids, and with the discipline of invalidating ResponseCache and SemanticCache on every publication, because a cache out of sync with the documentation is a silent generator of stale answers. The three pieces are now on the table: we know how to embed and compare (04-01), how to store and retrieve with filters (04-02) and how to populate the store with quality material (04-03). What remains is to join them with the artifacts from the previous modules — SYSTEM_DOCUBOT and its <documentation> block that will finally fill itself, answer_prompt(), the robust answer() from 03-02 — into a single function that takes a question and returns a grounded answer with real sources. That is the module's culminating moment: building the complete RAG pipeline (04-04).

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved