The Project | About Us | Contribute | Donations | License

HOME

The integration from 03-01 works, but it has two weaknesses that separate a prototype from a usable system. The first one any Nubelia employee notices with their first question: DocuBot goes silent for several seconds and then dumps the answer all at once. The second one the team will notice on the worst possible day: when the API returns a 429 for rate limiting or a server error, the bot will simply blow up with a traceback. In this lesson we solve both: streaming so the response flows as it is generated, and a complete strategy of errors, retries with backoff and graceful degradation so DocuBot fails well when something fails — because something, sooner or later, will fail.

Why streaming: perceived latency
Consuming a stream in Python
Streaming and JSON outputs: validate at the end
Taxonomy of API errors
Retries with exponential backoff and jitter
Timeouts, idempotency and what NOT to retry
Circuit breaker: knowing when to stop insisting (concept)
Graceful degradation in DocuBot

Why streaming: perceived latency

An LLM generates the response token by token, but the call from 03-01 waits until all the tokens exist before returning anything. If the response takes 6 seconds to generate, the user stares at an empty screen for 6 seconds.

Streaming changes the contract: the server sends each chunk as soon as it generates it and the client consumes it on the fly. The total time is the same (tokens are generated at the same rate), but the perception is radically different:

Metric	Without streaming	With streaming
Time until the first visible text	6 s (all or nothing)	~0.5–1.5 s (first token)
Time until the complete response	6 s	6 s (the same)
User's impression	"It's hung"	"It's typing"
Ability to abort a response going wrong	No	Yes, midway

For DocuBot's chat, streaming is mandatory: a support agent with a customer on the phone will not tolerate staring at a spinner. The general rule: conversational interface → streaming; batch process (classifying 500 tickets overnight) → no streaming, because there nobody is watching the screen.

There is a second, more technical reason: for long responses, keeping an HTTP connection open waiting for the complete response ends up colliding with intermediate network timeouts. Streaming's constant trickle keeps the connection alive.

Consuming a stream in Python

The SDK exposes a context manager that iterates over text chunks. The most direct form:

# streaming.py
from client import client, MODEL, extract_text
from prompts import SYSTEM_DOCUBOT, answer_prompt

def stream_response(question: str, documentation: str) -> str:
    """Prints the response as it is generated and returns the full text."""
    with client.messages.stream(
        model=MODEL,
        max_tokens=1024,
        system=SYSTEM_DOCUBOT,
        messages=[{"role": "user", "content": answer_prompt(question, documentation)}],
    ) as stream:
        chunks = []
        for text in stream.text_stream:
            print(text, end="", flush=True)   # show it to the user NOW
            chunks.append(text)               # and accumulate for later processing

        # Once the stream is exhausted we can ask for the complete final message,
        # with usage and stop_reason just like a normal call.
        final = stream.get_final_message()

    print()  # newline when done
    print(f"[output tokens: {final.usage.output_tokens}, stop: {final.stop_reason}]")
    return "".join(chunks)

Key points in the code:

client.messages.stream(...) accepts the same parameters as messages.create(); only how the response is consumed changes.
stream.text_stream is an iterator of text chunks. Each iteration delivers one or a few tokens; flush=True forces them to be painted immediately on the console (without it, Python may hold the output in a buffer and ruin the effect).
We accumulate the chunks in a list and join them at the end: displaying and processing are two different needs, and the "print and accumulate" pattern covers both.
stream.get_final_message() returns, once the stream has finished, the same response object you already know from 03-01: usage, stop_reason, content. Streaming does not cost you the telemetry.

Under the hood, the provider sends typed events (message start, block start, text delta, block end, message end) over a server-sent events (SSE) connection. text_stream is the convenient shortcut that filters only the text deltas; if some day you need the raw events (for instance, to distinguish block types), the same stream lets you iterate them one by one. The concept is identical in every provider: OpenAI and Gemini also emit incremental chunks consumed in a loop.

Streaming and JSON outputs: validate at the end

Here a practical limitation appears that matters for DocuBot: our 02-03 contract requires a complete JSON, and a partial JSON is not parseable. {"category": "billing", "answ cannot be validated; only the whole document can.

The design consequence:

You may display the chunks as they arrive (perceived latency).
You must validate only at the end, with the full accumulated text and the 02-03 parser.

import json
from prompts import FALLBACK_RESPONSE
from validation import clean_json, validate_contract

def stream_docubot(question: str, documentation: str) -> dict:
    text = stream_response(question, documentation)  # displays live and accumulates
    try:
        return validate_contract(json.loads(clean_json(text)))
    except (json.JSONDecodeError, TypeError):
        return FALLBACK_RESPONSE

This creates a user-experience tension: if the output is the raw JSON, the user would see braces and quotes flowing across the screen. The usual solutions, from simplest to most elaborate:

Don't stream on the JSON routes and reserve streaming for free-text responses. Simple and sufficient for many cases.
Internal streaming, render at the end: you consume the stream (the connection stays alive, you can measure the first token) but only show the user a progress indicator until validation.
Extract the answer field live with an incremental parser that detects when it is "inside" that field's value and emits only that. It is fragile and rarely worth it; bring it up in a design review before attempting it.

For DocuBot we adopt option 2: it keeps the contract intact without giving up the live connection.

Taxonomy of API errors

Every LLM API returns HTTP errors with the same underlying logic: 4xx = the problem is yours, 5xx = the problem is theirs. The ones you will see in practice:

Code	Name	Typical cause	Retry?
400	Invalid request	Malformed parameter; input exceeding the context window	❌ No: it will fail again. Fix the request (or trim the context, 03-03)
401	Authentication	Key missing, mistyped or revoked	❌ No: check the environment variable
403	Permissions	The key has no access to that model or feature	❌ No: check the plan/permissions
404	Not found	Nonexistent model identifier (typo)	❌ No: fix the name
413	Request too large	Request body exceeds limits	❌ No: reduce the input
429	Rate limit	Too many requests or tokens per minute	✅ Yes, with a wait (honoring `retry-after` if present)
500	Internal error	Transient provider failure	✅ Yes, with backoff
529	Overloaded	The service is saturated (equivalent to "come back in a while")	✅ Yes, with more generous backoff

And one more row that isn't HTTP: connection errors (DNS, network timeout, connection dropped mid-stream). There is no code because the request never arrived or never came back; they are retryable too.

The SDK translates these codes into typed exceptions, which allows a readable try/except instead of comparing numbers:

import anthropic

try:
    response = client.messages.create(...)
except anthropic.BadRequestError as e:      # 400: context exceeded, invalid parameter
    print(f"Invalid request: {e.message}")
except anthropic.AuthenticationError:        # 401
    print("Invalid API key: check ANTHROPIC_API_KEY")
except anthropic.RateLimitError as e:        # 429
    wait = int(e.response.headers.get("retry-after", "60"))
    print(f"Rate limit hit; the server suggests waiting {wait}s")
except anthropic.APIStatusError as e:        # remaining HTTP codes
    if e.status_code >= 500:
        print(f"Provider error ({e.status_code}): retryable")
    else:
        print(f"Client error ({e.status_code}): {e.message}")
except anthropic.APIConnectionError:         # the network, not the API
    print("Network error: retryable")

Note the order: from the most specific exception to the most general. RateLimitError before APIStatusError, because the former is conceptually a subtype of the latter and a generic except placed first would swallow it. This exception-hierarchy pattern exists in every SDK from every provider under different names; the 4xx/5xx/network taxonomy is universal.

The 400 for context exceeded deserves a special mention: today it is unlikely (we send very little), but in 03-03, when we accumulate conversation history, it will become a structural risk. Remember it.

Retries with exponential backoff and jitter

Faced with a 429 or a 5xx, retrying immediately is counterproductive: if the server is saturated, a thousand clients retrying at once saturate it further. The standard strategy has two ingredients:

Exponential backoff: wait longer and longer between attempts (1s, 2s, 4s, 8s...). It gives the transient problem time to resolve.
Jitter (randomness): add a random component to each wait. Without it, all the clients that failed at the same time retry at the same time, creating synchronized waves of traffic (the "thundering herd" problem).

Complete implementation for DocuBot:

# retries.py
import random
import time
import anthropic

from client import client

def call_with_retries(
    max_retries: int = 4,
    base_wait: float = 1.0,
    max_wait: float = 30.0,
    **kwargs,
):
    """Sends a request to the API with exponential backoff + jitter.

    Retries ONLY transient errors (429, 5xx, network). Client errors
    (400, 401, 403, 404...) are propagated immediately.
    """
    last_exception = None

    for attempt in range(max_retries):
        try:
            return client.messages.create(**kwargs)

        except anthropic.RateLimitError as e:
            last_exception = e
            # If the server dictates how long to wait, obey
            retry_after = e.response.headers.get("retry-after")
            if retry_after is not None:
                wait = min(float(retry_after), max_wait)
            else:
                wait = None  # compute with backoff below

        except anthropic.APIStatusError as e:
            if e.status_code < 500:
                raise          # 4xx other than 429: retrying won't fix it
            last_exception = e
            wait = None

        except anthropic.APIConnectionError as e:
            last_exception = e
            wait = None

        if wait is None:
            # exponential backoff: base * 2^attempt, capped, + jitter
            wait = min(base_wait * (2 ** attempt), max_wait)
            wait += random.uniform(0, base_wait)

        print(f"[attempt {attempt + 1}/{max_retries} failed; waiting {wait:.1f}s]")
        time.sleep(wait)

    # Attempts exhausted: propagate the last error so the caller can decide
    raise last_exception

Design decisions you must be able to justify:

Only transient failures are retried. A 401 retried 4 times is 4 identical failures, just slower. The early raise on non-429 4xx errors is as important as the loop.
retry-after rules. When the server tells you how long to wait, your heuristic is redundant.
The wait is capped (max_wait): without a ceiling, attempt 6 would wait 32s, attempt 8 more than 2 minutes, and the user will be long gone.
The last error is propagated. The retry function does not decide what to show the user; that belongs to the degradation layer (last section).

Important: official SDKs already ship configurable automatic retries — in Anthropic's, anthropic.Anthropic(max_retries=4) retries 429/5xx/network with backoff for you, and other providers offer the equivalent. In production, use the SDK's first and write your own only if you need behavior it doesn't cover (your own logging, metrics, per-query-type policies). We implemented it by hand because understanding what the SDK does internally is what lets you configure it with judgment and debug it when something doesn't add up.

Timeouts, idempotency and what NOT to retry

Timeouts. A timeout limits how long you wait for a response before giving up. Without one, a hung connection blocks your process indefinitely. The client lets you configure it globally or per call:

client = anthropic.Anthropic(timeout=30.0)  # seconds, for all calls

# or one-off, without touching the global client:
client.with_options(timeout=10.0).messages.create(...)

Criterion: the timeout must be consistent with the use case. The classifier (20 output tokens) should not take more than a few seconds: short timeout. A long streamed response can legitimately take longer: a more generous timeout, or streaming, which spreads out the wait. Keep in mind that an expired timeout usually counts as a retryable error, so the worst-case total time is approximately timeout × (retries + 1) — size both together.

Idempotency. An operation is idempotent if running it twice produces the same effect as running it once. DocuBot's calls (question → answer) are: retrying is safe, at worst you pay for the generation twice. But watch out for the general pattern: if your LLM call is coupled to a side effect ("generate the text AND send the email", "answer AND deduct credit from the customer"), a retry can duplicate the effect. Design rule: separate the generation (retryable) from the side effect (execute exactly once, after validating). In module 5, when the LLM calls tools that modify state, this rule will be critical.

What not to retry, in summary:

4xx errors other than 429 (they will fail again).
Requests whose content violated a provider policy (it will fail again and pollute your metrics).
Non-idempotent operations without protection (risk of a duplicated effect).
And don't retry infinitely: every retry loop needs a limit and an exit toward degradation.

Circuit breaker: knowing when to stop insisting

Retries solve isolated failures. But if the provider has been down for 10 minutes, every request from every user will go through its full 4-attempt cycle with waits — multiplying latency and useless traffic exactly when the other system is at its worst.

The circuit breaker pattern adds shared memory about the service's health. Conceptually, three states:

Closed (normal): requests pass through. Recent failures are counted.
Open: after crossing a failure threshold (e.g. 5 failures in 60 seconds), the circuit "trips": during a cooldown period, requests are not even attempted — they go straight to the degraded response. Failing fast is the goal.
Half-open: once the cooldown expires, one probe request is let through. If it works, the circuit closes (back to normal); if it fails, it reopens.

We will not implement it here (libraries like pybreaker ship it ready-made, and the observability that feeds it is covered in 06-04); what you should retain is the division of responsibilities: retries for the isolated failure, circuit breaker for the sustained outage. A mature system has both.

Graceful degradation in DocuBot

Everything above converges on a product question: what does the Nubelia employee see when, despite everything, the call fails? A traceback is unacceptable; silence, almost worse. We already designed the answer in 02-03: FALLBACK_RESPONSE, an object that satisfies the JSON contract with conservative values. Now we give it its definitive role as the last link in the chain:

# docubot.py — the layer that ties together all the pieces of the module
import json
import anthropic

from client import MODEL, extract_text
from retries import call_with_retries
from prompts import SYSTEM_DOCUBOT, answer_prompt, FALLBACK_RESPONSE
from validation import clean_json, validate_contract

def answer(question: str, documentation: str) -> dict:
    """Full pipeline: call with retries -> validation -> fallback.

    Whatever happens, it ALWAYS returns a dict that satisfies the contract.
    """
    try:
        response = call_with_retries(
            model=MODEL,
            max_tokens=1024,
            system=SYSTEM_DOCUBOT,
            messages=[{"role": "user", "content": answer_prompt(question, documentation)}],
        )
    except anthropic.APIError:
        # Retries exhausted or non-retryable error: degrade, don't blow up.
        # (In 06-04 we will add logging and metrics here.)
        return FALLBACK_RESPONSE

    if response.stop_reason == "max_tokens":
        return FALLBACK_RESPONSE

    try:
        return validate_contract(json.loads(clean_json(extract_text(response))))
    except (json.JSONDecodeError, TypeError):
        return FALLBACK_RESPONSE

The guarantee this function offers is the one that defines a robust system: its return type does not depend on the world working. Network down, provider saturated, corrupt JSON, truncated response — the caller always receives a valid contract, and the interface always has something dignified to show ("I can't check the documentation right now; please try again in a few minutes"). The details of how many times it degraded and why belong to observability (06-04); the fact that it never blows up belongs to this lesson.

Common Mistakes and Tips

Retrying 400s. The most expensive rookie mistake: a loop retrying an invalid request 4 times with 30s waits turns a one-second bug into minutes of latency. 4xx (except 429) → fix, don't insist.
Backoff without jitter. It works in your test with a single client and fails in production with a hundred: they all retry in sync. The random.uniform is not decorative.
Ignoring retry-after. The server is telling you exactly how long to wait; your exponential is a guess.
Validating JSON chunk by chunk while streaming. A partial JSON never parses; accumulate and validate at the end. If you see constant JSONDecodeErrors on the streaming path, it's almost certainly this.
Forgetting flush=True when printing the stream: the output buffer holds the chunks and the "typewriter effect" disappears, so streaming loses its visual raison d'être.
Doubling up retries (the SDK's enabled AND yours wrapped around them): 4 × 4 = 16 real attempts. Choose one layer and disable or limit the other (max_retries=0 if your function is in charge).
Tip: trigger the errors in development, don't wait for production. A fake key gives you the 401; max_tokens=10 gives you the truncation; a timeout=0.001 gives you the connection error. Every branch of the except should have executed at least once before your eyes.
Tip: always record the request identifier (response.id or the one in the exception) when reporting errors to the provider: it is what allows them to trace your case.

Exercises

Exercise 1

Write stream_with_metrics(question, documentation) that consumes the stream and returns a dictionary with: text (complete), t_first_token (seconds from sending until the first chunk) and t_total (seconds until the end). Use time.perf_counter(). Run the same question with and without streaming and compare which metric improves and which does not.

Exercise 2

Modify call_with_retries() so it accepts an on_failure parameter (optional callable) invoked on every failed attempt with (attempt, exception, wait). Use it to print a readable trace. Then trigger a 401 with a fake key and verify that it is not retried (the trace must not appear).

Exercise 3

Design (in pseudocode or Python) a minimal circuit breaker for DocuBot: a CircuitBreaker class with failure_threshold=5, window_seconds=60 and cooldown_seconds=120, and methods allow() -> bool, record_success() and record_failure(). Integrate it conceptually into answer(): where is it consulted and where is it recorded?

Solutions

Solution 1:

import time

def stream_with_metrics(question, documentation):
    start = time.perf_counter()
    t_first_token = None
    chunks = []

    with client.messages.stream(
        model=MODEL, max_tokens=1024, system=SYSTEM_DOCUBOT,
        messages=[{"role": "user", "content": answer_prompt(question, documentation)}],
    ) as stream:
        for text in stream.text_stream:
            if t_first_token is None:
                t_first_token = time.perf_counter() - start
            chunks.append(text)

    return {
        "text": "".join(chunks),
        "t_first_token": t_first_token,
        "t_total": time.perf_counter() - start,
    }

You will observe that t_total is similar with and without streaming, but t_first_token is a small fraction of it. That difference is the perception improvement — and in 03-04 we will call it by its name (TTFT) and measure it systematically.

Solution 2:

def call_with_retries(max_retries=4, base_wait=1.0,
                      max_wait=30.0, on_failure=None, **kwargs):
    last_exception = None
    for attempt in range(max_retries):
        try:
            return client.messages.create(**kwargs)
        except anthropic.RateLimitError as e:
            last_exception = e
        except anthropic.APIStatusError as e:
            if e.status_code < 500:
                raise                      # this is where the 401 dies: no retry
            last_exception = e
        except anthropic.APIConnectionError as e:
            last_exception = e

        wait = min(base_wait * (2 ** attempt), max_wait)
        wait += random.uniform(0, base_wait)
        if on_failure:
            on_failure(attempt + 1, last_exception, wait)
        time.sleep(wait)
    raise last_exception

# Usage:
# call_with_retries(on_failure=lambda i, e, w: print(f"[{i}] {type(e).__name__}, waiting {w:.1f}s"), ...)

With the fake key, AuthenticationError (which in practice inherits from the status error with a code < 500 in the SDK; if your version exposes it as its own class, add its except with a raise) propagates on the first attempt: zero traces, zero waits. Exactly the correct behavior.

Solution 3:

import time

class CircuitBreaker:
    def __init__(self, failure_threshold=5, window_seconds=60, cooldown_seconds=120):
        self.threshold = failure_threshold
        self.window = window_seconds
        self.cooldown = cooldown_seconds
        self.failures = []          # timestamps of recent failures
        self.open_until = 0.0       # instant when the cooldown ends

    def allow(self) -> bool:
        return time.monotonic() >= self.open_until

    def record_success(self):
        self.failures.clear()

    def record_failure(self):
        now = time.monotonic()
        self.failures = [t for t in self.failures if now - t < self.window]
        self.failures.append(now)
        if len(self.failures) >= self.threshold:
            self.open_until = now + self.cooldown
            self.failures.clear()

Integration into answer(): it is consulted at the beginning (if not breaker.allow(): return FALLBACK_RESPONSE — fail fast, without touching the network) and recorded at the end of the call (success after the create, failure in the except). The half-open state emerges on its own: once the cooldown passes, allow() returns True again and the next request acts as the probe.

Conclusion

DocuBot now behaves like a system and not like a script: streaming makes generation visible from the first token (validating the JSON only at the end, with the 02-03 parser intact), the error taxonomy separates what you fix (4xx) from what you wait out (429, 5xx, network), retries with exponential backoff and jitter absorb the isolated failures — knowing the SDK ships its own configurable ones —, timeouts and idempotency mark the boundaries of what is retryable, the circuit breaker is noted down for sustained outages, and FALLBACK_RESPONSE guarantees that, whatever happens, the Nubelia employee receives a dignified answer and not a traceback. But our DocuBot still has the memory of a goldfish: every call starts from scratch, and if an agent asks "and how do I configure that?", the bot has no idea what "that" is. In the next lesson we tackle conversation: why LLMs remember nothing between calls, how the illusion of memory is built by sending the full history, and what strategies exist so that history doesn't devour the context window or the budget.

Streaming, Errors and Retries

Contents

Why streaming: perceived latency

Consuming a stream in Python

Streaming and JSON outputs: validate at the end

Taxonomy of API errors

Retries with exponential backoff and jitter

Timeouts, idempotency and what NOT to retry

Circuit breaker: knowing when to stop insisting

Graceful degradation in DocuBot

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Generative AI and LLMs for Developers Course

Module 1: Generative AI Fundamentals

Module 2: Prompt Engineering

Module 3: Integrating LLMs into Applications

Module 4: RAG - Retrieval-Augmented Generation

Module 5: Function Calling and Agents

Module 6: Toward Production