The Project | About Us | Contribute | Donations | License

HOME

DocuBot answers, streams and survives errors — but it forgets each question as soon as it answers it. If a support agent asks "how do I export a project to CSV?" and then "and is that available on the Starter plan?", the bot has no idea what "that" is. In this lesson we build the conversation layer: we will understand why LLMs have no memory and how the illusion of continuity is manufactured by resending the full history on every call, implement a Conversation class in Python, measure the problem this creates (a growing history devours context window and budget) and compare the strategies to contain it. We will close with the jump to multi-user: isolating each Nubelia employee's conversation.

LLMs have no memory: the illusion of conversation
The Conversation class in Python
The history grows: window limits and rising cost
History management strategies
Multi-user: isolated sessions and simple persistence
When what's missing isn't history: a preview of RAG

LLMs have no memory: the illusion of conversation

Let's recover the limitation we catalogued in 01-04: an LLM is stateless. The API stores nothing between calls: not who you are, not what you asked ten seconds ago. Every request is, to the model, the first and only one.

So how is it that any chat with an LLM "remembers" what you said? Because the client resends the entire conversation on every call. The memory isn't in the model; it's in your messages list:

# Call 1: the model sees ONE question
messages = [
    {"role": "user", "content": "How do I export a project to CSV?"},
]

# Call 2: the model sees the WHOLE conversation, again from scratch
messages = [
    {"role": "user", "content": "How do I export a project to CSV?"},
    {"role": "assistant", "content": "Go to Project > Settings > Export > CSV..."},
    {"role": "user", "content": "And is that available on the Starter plan?"},
]

On the second call, the model can resolve "that" because the previous turn travels in the request. It doesn't remember it: it rereads it. This is where the user/assistant roles we defined in 02-01 finally click into place — the assistant role exists precisely to re-inject what the model itself said earlier.

Three immediate consequences of this design:

Every call pays for the whole conversation as input tokens. Turn 20 resends (and bills) the previous 19.
The conversation is yours. The provider does not keep it for you as far as the API is concerned: if your process dies and you didn't persist it, it never existed.
You can edit the past. Since the history is a list you build, you can shorten it, summarize it or filter it. This flexibility is exactly what we will exploit in the strategies section.

The `Conversation` class in Python

We encapsulate the pattern in a reusable class. Design decisions: the system prompt stays outside the history (it travels in its own parameter, it is not one more turn), and the class reuses the robust pipeline from 03-02:

# conversation.py
import anthropic

from client import MODEL, extract_text
from retries import call_with_retries
from prompts import SYSTEM_DOCUBOT

class Conversation:
    """Multi-turn conversation with an LLM.

    Keeps the user/assistant history and resends it in full on every
    call. The system prompt is kept separate: it is not part of the
    message list.
    """

    def __init__(self, system: str = SYSTEM_DOCUBOT,
                 model: str = MODEL, max_tokens: int = 1024):
        self.system = system
        self.model = model
        self.max_tokens = max_tokens
        self.messages: list[dict] = []      # only user/assistant turns
        self.total_input_tokens = 0         # running accounting
        self.total_output_tokens = 0

    def send(self, user_text: str) -> str:
        """Appends the user's turn, calls the API and records the answer."""
        self.messages.append({"role": "user", "content": user_text})

        try:
            response = call_with_retries(
                model=self.model,
                max_tokens=self.max_tokens,
                system=self.system,
                messages=self.messages,      # the WHOLE history, every time
            )
        except anthropic.APIError:
            # Don't leave the history unbalanced: remove the failed turn
            self.messages.pop()
            raise

        text = extract_text(response)

        # The assistant's turn MUST be saved: it's what gives it memory
        self.messages.append({"role": "assistant", "content": text})

        self.total_input_tokens += response.usage.input_tokens
        self.total_output_tokens += response.usage.output_tokens
        return text

    @property
    def num_turns(self) -> int:
        """One turn = one question/answer pair."""
        return len(self.messages) // 2

Details that matter:

The pop() in the except. If the call fails, the user turn had already been appended. Leaving it there would produce two consecutive user messages with no answer in between on the next attempt — a corrupted history. Undoing on error preserves the invariant "user and assistant alternate".
Saving the assistant's answer is not optional. If you forget the assistant append, the model never sees its own previous answers and the "memory" silently disappears — the most common bug in this layer.
The token accounting (total_input_tokens, total_output_tokens) looks like bureaucracy now; in the next section we will use it to see the growth problem, and in 03-04 to put a price on it.

Usage:

chat = Conversation()
print(chat.send("How do I export a project to CSV?"))
print(chat.send("And is that available on the Starter plan?"))   # "that" now resolves
print(f"Turns: {chat.num_turns}, accumulated input tokens: {chat.total_input_tokens}")

The history grows: window limits and rising cost

The illusion of memory has a compounding price. Assume fictional but realistic numbers for DocuBot: system prompt ~600 tokens, each question ~50 tokens, each answer ~250 tokens. The input of every call is system + the whole history + the new question:

Turn	Prior history (tokens)	Input of this call	Accumulated input paid
1	0	~650	~650
5	~1,200	~1,850	~6,400
10	~2,700	~3,350	~19,500
25	~7,200	~7,850	~106,000
50	~14,700	~15,350	~400,000

Two readings of the table:

The input per call grows linearly with the turns... but the accumulated spend grows quadratically, because every turn re-pays all the previous ones. A 50-turn conversation doesn't cost 50 times the first call: it costs hundreds of times more in input tokens.
The context window is finite (01-04). When system + history + question + room for the answer exceeds the model's window, the API will return the 400 for context exceeded we catalogued in 03-02. With large windows it will take a while to arrive, but it is a deterministic wall: without management, every sufficiently long conversation dies there.

And there is a third, subtler problem you already know from 01-04: "lost in the middle". Even when it fits, a huge history dilutes attention — the important detail buried in turn 12 of 40 carries less weight than what sits at the beginning or the end. More history is not always better memory; past a certain point it is worse signal.

Conclusion: the Conversation class needs a history management policy. It is not an optimization — it is survival.

History management strategies

Strategy 1: truncation by turns

The simplest: keep only the last N complete turns.

def truncate_by_turns(messages: list[dict], max_turns: int = 10) -> list[dict]:
    """Keeps the last max_turns user/assistant pairs.

    ALWAYS cut at a pair boundary: the first message of the result
    must be a 'user' (the API requires starting with user).
    """
    max_messages = max_turns * 2
    if len(messages) <= max_messages:
        return messages
    trimmed = messages[-max_messages:]
    # Guarantee it starts with 'user' (in case the cut fell mid-pair)
    while trimmed and trimmed[0]["role"] != "user":
        trimmed = trimmed[1:]
    return trimmed

It is applied inside send(), right before the call: messages=truncate_by_turns(self.messages). Note that we truncate what is sent, not necessarily what is stored — you can keep the full history locally (for auditing or for the interface) and send only the tail.

Advantage: trivial, bounded cost, zero extra calls. Drawback: the memory is deferred amnesia — whatever was said 11 turns ago vanishes completely, even if it was the key fact ("my company uses the Business plan").

Strategy 2: summarizing the history with the LLM itself

When the history exceeds a threshold, the model itself is asked to compress the old turns into a summary, and that summary replaces the turns it summarizes:

SUMMARY_PROMPT = """Summarize the following support conversation in fewer than 150
words. Preserve: the user's concrete details (plan, configuration, project
names), the problem being addressed and what has already been resolved. Omit pleasantries.

<conversation>
{conversation}
</conversation>"""

def summarize_history(messages: list[dict], keep_turns: int = 4) -> list[dict]:
    """Compresses the old turns into a summary; keeps the recent ones as-is."""
    limit = keep_turns * 2
    if len(messages) <= limit:
        return messages

    old, recent = messages[:-limit], messages[-limit:]
    text = "\n".join(f"{m['role']}: {m['content']}" for m in old)

    response = call_with_retries(
        model=MODEL,
        max_tokens=300,
        messages=[{"role": "user", "content": SUMMARY_PROMPT.format(conversation=text)}],
    )
    summary = extract_text(response)

    # The summary is injected as a synthetic user/assistant pair at the start
    return [
        {"role": "user", "content": f"[Summary of the previous conversation]\n{summary}"},
        {"role": "assistant", "content": "Understood, I have the context of the previous conversation."},
    ] + recent

Notice the summary prompt: it says what to preserve (concrete details, decisions), not just "summarize". A generic summary loses exactly what mattered. Even so, the summary is lossy: it is the LLM itself deciding what deserves to survive, and sometimes it gets it wrong. It also costs an extra call every time it compacts (which is why it is done on a threshold, not on every turn).

Strategy 3: sliding window (hybrid)

In practice, the most common configuration combines the previous two: an accumulated summary at the beginning (long-term memory, compressed) + the last N literal turns (short-term memory, intact). Every time the recent history overflows, the oldest turns are merged into the summary — the window "slides", leaving a summarized sediment behind it. It is strategy 2 applied incrementally, with the structure of strategy 1.

Comparison

Strategy	Extra cost	Memory fidelity	Complexity	When to use it
Send everything (no management)	Grows without bound	Perfect until hitting the window	None	Prototypes; conversations guaranteed short
Truncation by turns	None	Zero beyond N turns	Very low	Mostly independent queries; DocuBot v1
LLM summarization	One call per compaction	High but lossy (the LLM decides)	Medium	Long conversations where old context matters
Sliding window	One call per slide	Recent perfect + old compressed	Medium-high	Long-session assistants; the production default

For DocuBot we choose truncation at 10 turns as the initial policy: support queries usually resolve in a few turns and simplicity wins. The decision is documented and reversible — if the 02-04 evaluation (which we can now run against real conversations) shows context loss, the sliding window is the next step. A note for 03-04: the history strategy is also a cost lever, and summarization will play a second role there.

Multi-user: isolated sessions and simple persistence

So far, a single conversation. But DocuBot will serve all of Nubelia's employees at once, and this imposes an absolute rule: one user's conversation must never contaminate another's. Anna asking about permissions must not cause Berta to get answers about permissions — and even worse would be leaking a customer's information between sessions (privacy, 01-04 and module 6).

The minimal solution: a dictionary of sessions indexed by a user/session identifier, with lazy creation:

# sessions.py
import json
import pathlib

class SessionManager:
    """Isolates one Conversation per user/session. Simple JSON persistence."""

    def __init__(self, directory: str = "./sessions"):
        self.sessions: dict[str, Conversation] = {}
        self.directory = pathlib.Path(directory)
        self.directory.mkdir(exist_ok=True)

    def conversation(self, session_id: str) -> Conversation:
        """Returns the session's conversation, creating or loading it if needed."""
        if session_id not in self.sessions:
            self.sessions[session_id] = self._load(session_id) or Conversation()
        return self.sessions[session_id]

    def ask(self, session_id: str, text: str) -> str:
        conv = self.conversation(session_id)
        answer = conv.send(text)
        self._save(session_id, conv)      # persist after every turn
        return answer

    # --- simple persistence: one JSON file per session ---

    def _path(self, session_id: str) -> pathlib.Path:
        safe = "".join(c for c in session_id if c.isalnum() or c in "-_")
        return self.directory / f"{safe}.json"

    def _save(self, session_id: str, conv: Conversation):
        self._path(session_id).write_text(
            json.dumps(conv.messages, ensure_ascii=False), encoding="utf-8"
        )

    def _load(self, session_id: str) -> "Conversation | None":
        path = self._path(session_id)
        if not path.exists():
            return None
        conv = Conversation()
        conv.messages = json.loads(path.read_text(encoding="utf-8"))
        return conv

Usage — two fictional employees, zero interference:

manager = SessionManager()
manager.ask("employee-anna", "How do I give a collaborator edit permissions?")
manager.ask("employee-bruno", "How do I connect the calendar?")
manager.ask("employee-anna", "And how do I take them away?")   # resolves with HER history

Honest scope notes:

The session identifier must come from real authentication (Nubelia's SSO, the web session), never from a field the user types. A tamperable session_id is access to someone else's history.
The one-file-per-session JSON persistence is didactic and sufficient for a pilot; in production, with real concurrency, this is a database (even if it's SQLite) plus a session expiry/deletion policy — support history contains internal information and the privacy rules of module 6 (06-02) apply to it.
The dict lives in the memory of one process. With multiple processes or servers, shared persistence stops being optional: it is the only real state.

When what's missing isn't history: a preview of RAG

Let's close with a conceptual distinction that organizes the whole course. In this lesson we have managed one kind of context: what has been said in this conversation. But when DocuBot cannot answer, it is almost never because it is missing an old turn — it is because it is missing knowledge: the right fragment of Nubelia's documentation.

The naive temptation would be to "stuff more things into the context": all the documentation in the system prompt, endless histories. You already know why that fails — finite window, quadratic cost, lost in the middle. The correct answer is different: for each question, search for only the relevant fragments of the documentation and inject them into the call. That is retrieval-augmented generation (RAG), and it is exactly module 4: embeddings and semantic search (04-01), vector databases (04-02), ingestion and chunking (04-03) and the complete pipeline (04-04). The mnemonic rule: history for what was said, RAG for what is known. Today you built one half of the equation; the other half starts in the next module — but first, a mandatory stop: knowing how much all of this costs.

Common Mistakes and Tips

Forgetting to save the assistant's turn. The symptom is subtle: nothing fails, but the bot "doesn't register" its own previous answers. Verify that the history alternates user/assistant and grows two by two.
Putting the system prompt in as the first user message. It half-works and degrades rule compliance: the system prompt has its own channel (02-01) and that is where it must go. Besides, mixed into the history it would eventually be truncated by your own management policy.
Truncating mid-pair. A history that starts with assistant or has two consecutive user messages causes API errors or erratic behavior. Always truncate at a complete-turn boundary.
Leaving the history corrupted after an error. If the call fails after the user turn's append, remove it (the class's pop()). An unbalanced history today = a weird error tomorrow.
Sharing a global Conversation across users "for now". It is the easiest context leak (and potentially data leak) to create and the hardest to detect. Per-session isolation from day one.
Tip: log usage.input_tokens per turn (the class already accumulates it). If it grows linearly turn after turn, your management policy isn't working; with truncation active it should stabilize into a plateau.
Tip: when debugging "the bot forgot X", print exactly the messages list that was sent in that call. The answer is always there: either X no longer travels (aggressive truncation/summarization), or it travels buried in the middle of an enormous history (lost in the middle).

Exercises

Exercise 1

Add a parameter max_sent_turns: int | None = None to Conversation. If it is not None, send() must run what is sent to the API through truncate_by_turns(), while keeping self.messages complete in memory. Also add a messages_sent_last_call property so you can inspect what the model actually saw.

Exercise 2

Write a simulation (no real API: use estimated lengths, 1 token ≈ 0.75 words as in 01-02) that compares the accumulated input tokens after 30 turns under three policies: no management, truncation at 10 turns, and sliding window (a 200-token summary + 4 literal turns). Print a table.

Exercise 3

Design the session-isolation test case: using SessionManager, session A establishes a fact ("I'm on the Business plan") and session B then asks "what plan am I on?". Define DocuBot's correct behavior in B according to the 02-01 system prompt and write the assertion (hint: the exact anti-hallucination phrase from 02-01 is detectable).

Solutions

Solution 1:

class Conversation:
    def __init__(self, system=SYSTEM_DOCUBOT, model=MODEL,
                 max_tokens=1024, max_sent_turns=None):
        # ... as before ...
        self.max_sent_turns = max_sent_turns
        self._last_sent: list[dict] = []

    def send(self, user_text: str) -> str:
        self.messages.append({"role": "user", "content": user_text})
        to_send = self.messages
        if self.max_sent_turns is not None:
            to_send = truncate_by_turns(self.messages, self.max_sent_turns)
        self._last_sent = to_send
        # ... call with messages=to_send, rest unchanged ...

    @property
    def messages_sent_last_call(self):
        return self._last_sent

The separation "full history in memory / trimmed history in the request" is the right pattern: the interface can keep showing the whole conversation even though the model only sees the tail.

Solution 2:

T_SYSTEM, T_QUESTION, T_ANSWER, T_SUMMARY = 600, 50, 250, 200
TURN = T_QUESTION + T_ANSWER   # 300 tokens per pair

def input_tokens(turn: int, policy: str) -> int:
    if policy == "no_management":
        history = (turn - 1) * TURN
    elif policy == "truncate_10":
        history = min(turn - 1, 10) * TURN
    elif policy == "sliding":
        recent = min(turn - 1, 4) * TURN
        history = recent + (T_SUMMARY if turn - 1 > 4 else 0)
    return T_SYSTEM + history + T_QUESTION

for policy in ("no_management", "truncate_10", "sliding"):
    total = sum(input_tokens(t, policy) for t in range(1, 31))
    print(f"{policy:>14}: accumulated input ~{total:,} tokens")

Approximate results: no management ~150,000 tokens; truncation ~90,000; sliding ~65,000 (plus the cost of the summary calls, which this simulation doesn't include — factor it in mentally when deciding). The gap explodes with more turns: at 100 turns, no management exceeds a million and the other two stay flat per turn.

Solution 3:

manager = SessionManager()

manager.ask("session-a", "I'm on the Business plan. Can I export to CSV?")
response_b = manager.ask("session-b", "What plan am I on?")

# Correct behavior: B doesn't have that fact in ITS history or in the
# provided documentation, and rule 3 of SYSTEM_DOCUBOT requires:
assert "I can't find that information in the available documentation." in response_b
assert "Business" not in response_b   # A's fact must NEVER leak

The second assertion is the important one: if "Business" appears in session B, you have a context leak between sessions — a privacy bug, not a quality bug. This case belongs in the permanent evaluation set (02-04, mild adversarial type) and we will meet it again in module 6.

Conclusion

DocuBot now converses: you know that an LLM's memory is an illusion manufactured by the client — every call resends the full history, with the system prompt kept apart —, you have the Conversation class that maintains that history with its invariants (role alternation, undo on error, token accounting), you have seen why it grows until it hits the context window and drives up the cost (linear per call, quadratic accumulated), and you have three measurable strategies to contain it — truncation, summarization with the LLM itself and the sliding window — plus the SessionManager that isolates and persists each Nubelia employee's conversation. And we planted the distinction that opens module 4: history for what was said, RAG for what is known. But before retrieving knowledge, we must settle the outstanding bill this lesson has been running up: every token of history gets paid for, and we still don't know how much. In the next lesson we put numbers on DocuBot — per-token prices, the monthly projection that refines that ~€400/month estimate from 01-04, the levers to lower it, latency as the second currency, and caching as the lever that attacks both.

Conversations and Context Management

Contents

LLMs have no memory: the illusion of conversation

The `Conversation` class in Python

The history grows: window limits and rising cost

History management strategies

Strategy 1: truncation by turns

Strategy 2: summarizing the history with the LLM itself

Strategy 3: sliding window (hybrid)

Comparison

Multi-user: isolated sessions and simple persistence

When what's missing isn't history: a preview of RAG

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Generative AI and LLMs for Developers Course

Module 1: Generative AI Fundamentals

Module 2: Prompt Engineering

Module 3: Integrating LLMs into Applications

Module 4: RAG - Retrieval-Augmented Generation

Module 5: Function Calling and Agents

Module 6: Toward Production

Conversations and Context Management

Contents

LLMs have no memory: the illusion of conversation

The Conversation class in Python

The history grows: window limits and rising cost

History management strategies

Strategy 1: truncation by turns

Strategy 2: summarizing the history with the LLM itself

Strategy 3: sliding window (hybrid)

Comparison

Multi-user: isolated sessions and simple persistence

When what's missing isn't history: a preview of RAG

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Generative AI and LLMs for Developers Course

Module 1: Generative AI Fundamentals

Module 2: Prompt Engineering

Module 3: Integrating LLMs into Applications

Module 4: RAG - Retrieval-Augmented Generation

Module 5: Function Calling and Agents

Module 6: Toward Production

The `Conversation` class in Python