The Project | About Us | Contribute | Donations | License

HOME

Up to now, everything DocuBot has "answered" we wrote ourselves on paper: prompts designed in module 2 and expected answers to evaluate against. In this lesson the system comes to life: we will set up a clean Python environment, make the first real call to an LLM API sending the system/user roles we designed in 02-01, dissect the response the provider returns (content, stop reason, token usage) and hook the defensive JSON parser from 02-03 up to a real response. We will use the Anthropic SDK as our example, but the concepts — messages with roles, output limit, usage object — are equivalent across providers, and we will close with a table that proves it.

From paper to the API: what we are going to build
Setting up the environment: venv, SDK and API key
The first call: system + user with the artifacts from prompts.py
Anatomy of the response: content, stop_reason and usage
Main parameters: model, max_tokens and temperature
Hooking the JSON parser from 02-03 up to a real response
Concept equivalence across providers

From paper to the API: what we are going to build

In 01-03 we made an architecture decision that now becomes real: managed API, mid-tier model and low temperature. That means we are not going to install or serve any model; we are going to send HTTP requests to a provider and receive responses. The provider's SDK handles transport, authentication and serialization, and we focus on what we already have: SYSTEM_DOCUBOT, the templates in prompts.py (version 1.2) and the JSON contract with its defensive validation.

The goal of this lesson is a client.py module that, given a question and a block of documentation, returns the validated dictionary {"category", "answer", "sources", "confidence"}. It is DocuBot's first "living" piece.

Setting up the environment: venv, SDK and API key

We will work in a virtual environment to isolate dependencies from the rest of the system:

# Create and activate the virtual environment
python -m venv .venv

# Linux / macOS
source .venv/bin/activate
# Windows (PowerShell)
.venv\Scripts\Activate.ps1

# Install the provider's SDK (example: Anthropic)
pip install anthropic

# Freeze dependencies for reproducibility
pip freeze > requirements.txt

The API key is never written into the code or pushed to the repository. It is defined as an environment variable, which is where the SDK looks for it automatically:

# Linux / macOS
export ANTHROPIC_API_KEY="your-key-here"

# Windows (PowerShell)
$env:ANTHROPIC_API_KEY = "your-key-here"

Good practices from day one:

Environment variable or secrets manager, never a string in the code. A key leaked in a commit is a security incident (and a billing one: whoever has it spends your quota).
Add .venv/ and any .env file to .gitignore.
Use different keys for development and production; that way you can revoke one without affecting the other.

The first call: system + user

We create client.py reusing the module 2 artifacts as they are. Notice that the system prompt travels in its own parameter, separate from the message list — exactly the role separation we studied in 02-01:

# client.py
import os
import anthropic

from prompts import SYSTEM_DOCUBOT, answer_prompt, PROMPT_VERSION

# The exact model identifier depends on the provider and the point in time;
# we read it from an environment variable with a placeholder as the default.
MODEL = os.environ.get("DOCUBOT_MODEL", "claude-...")

# The client reads ANTHROPIC_API_KEY from the environment: we never pass the key by hand.
client = anthropic.Anthropic()

def ask_docubot_raw(question: str, documentation: str):
    """Sends a question to the API and returns the full (unprocessed) response."""
    return client.messages.create(
        model=MODEL,
        max_tokens=1024,
        temperature=0.2,          # low: consistent answers (the 01-03 decision)
        system=SYSTEM_DOCUBOT,    # the v1.2 system prompt designed in 02-01
        messages=[
            {
                "role": "user",
                "content": answer_prompt(question, documentation),
            }
        ],
    )

Line-by-line breakdown:

anthropic.Anthropic() creates the client. We don't pass it the key: it resolves it from ANTHROPIC_API_KEY. If it doesn't exist, the call will fail with an authentication error — better that than a key written into the file.
model=MODEL says which model answers. We use a placeholder like "claude-..." because concrete identifiers change over time; in your project, pin the one you chose in 01-03.
system=SYSTEM_DOCUBOT sends DocuBot's identity, anti-hallucination rules and tone. In this provider it is a top-level parameter; in others it goes as one more message with the system role (we will see this in the final table).
messages is the list of conversation turns. For now a single user turn, whose content is generated by answer_prompt(): the employee's question plus the <documentation> block that rule 2 of the system prompt requires.
max_tokens=1024 is the output token cap: the response will never be longer than that. It is mandatory in this API and it is your first cost brake.

Let's try it with fictional Nubelia data:

if __name__ == "__main__":
    documentation = """## Exporting a project
To export a project to CSV: Project > Settings > Export > CSV.
The export includes tasks, assignees and dates. Available on
the Pro and Business plans."""

    question = "How do I export my project to CSV?"

    response = ask_docubot_raw(question, documentation)
    print(response)

If everything is configured correctly, running python client.py will show you, for the first time, a response generated by the model with your system prompt. It's worth pausing for a second: the rules you wrote in 02-01 are governing, right now, the output of a real model.

Anatomy of the response

The object the API returns contains much more than the text. Its important fields:

response = ask_docubot_raw(question, documentation)

# 1. The content: a LIST of blocks, not a plain string
for block in response.content:
    if block.type == "text":
        print(block.text)

# 2. Why it stopped generating
print(response.stop_reason)      # e.g. "end_turn"

# 3. How many tokens it consumed (this is the bill!)
print(response.usage.input_tokens)   # input tokens (system + user)
print(response.usage.output_tokens)  # generated tokens

# 4. Metadata
print(response.model)            # the model that actually answered
print(response.id)               # unique identifier of the request

Three details worth internalizing:

content is a list of blocks. Today it only contains text blocks, but the same field carries other types (tool calls in module 5, for example). Get used, from now on, to filtering by block.type == "text" instead of blindly accessing content[0].text.
stop_reason tells you why generation ended. The main values:

`stop_reason`	Meaning	What to do
`end_turn`	The model finished naturally	Normal case, process the response
`max_tokens`	The output cap ran out	The response is truncated: raise the cap or ask for shorter outputs
`stop_sequence`	A stop sequence you defined appeared	Process up to that point
`tool_use`	The model wants to call a tool	We will see this in module 5

For DocuBot, max_tokens is especially dangerous: a JSON cut off halfway will never validate. Checking stop_reason before parsing saves you from wasting time debugging "invalid JSON" when the real problem was the output cap.

usage is your cost meter. Input and output are billed separately and at different prices. In 03-04 we will build the cost estimate on top of these two numbers; for now, print them in every test to start building intuition about how much each call "weighs".

A helper function we will use throughout the course:

def extract_text(response) -> str:
    """Concatenates the text blocks of an API response."""
    return "".join(
        block.text for block in response.content if block.type == "text"
    )

Main parameters

The three parameters you will adjust constantly:

Parameter	What it controls	Guidance for DocuBot
`model`	Which model answers (capability, speed, price)	Mid-tier for answers; in 03-04 we will see the classifier can use a smaller one
`max_tokens`	Hard cap on output tokens	1024 is plenty for <150-word answers + JSON; it is a brake on cost and on runaway output
`temperature`	Sampling randomness (0 = very deterministic, 1 = very varied)	Low (0.0–0.3): DocuBot must be consistent, not creative

Two honest nuances about temperature:

Low temperature is not full determinism. As we saw in 01-04 (non-determinism), two identical calls can differ slightly even at temperature 0. It reduces the variance; it does not eliminate it.
Some recent models from some providers fix sampling internally and reject the temperature parameter with an invalid-request error. Check the documentation of the specific model you use; if it rejects it, simply omit the parameter and control style through the prompt.

Hooking the JSON parser from 02-03 up to a real response

In 02-03 we designed the contract {"category", "answer", "sources", "confidence"} and the defensive validation (clean_json, json.loads with try/except, conservative defaults, FALLBACK_RESPONSE). All of that was written against hypothetical responses; now we plug it into the real output:

import json
from prompts import FALLBACK_RESPONSE
from validation import clean_json, validate_contract  # the module written in 02-03

def ask_docubot(question: str, documentation: str) -> dict:
    """Full call: API -> text -> JSON validated against the 02-03 contract."""
    response = ask_docubot_raw(question, documentation)

    # A JSON truncated by max_tokens will never parse: detect it early and clearly.
    if response.stop_reason == "max_tokens":
        return FALLBACK_RESPONSE

    text = extract_text(response)
    try:
        data = json.loads(clean_json(text))   # strips ```json ... ``` and noise
    except (json.JSONDecodeError, TypeError):
        return FALLBACK_RESPONSE

    # Conservative defaults for missing or invalid fields (02-03)
    return validate_contract(data)

The first time you run this against the real model, you will likely discover things paper never showed: responses wrapped in ```json, a confidence field with a value outside the enumeration, a sources list arriving as a string. This is exactly why we built the defensive validation — and why clean_json and the defaults were never paranoia, but engineering.

Native structured output. As we announced in 02-03, providers offer modes that guarantee the output conforms to a JSON schema: in Anthropic you declare a schema in the request (output_config with json_schema), OpenAI has the equivalent with response_format/strict schemas, and Gemini with response_schema. When your provider and model support it, turn it on: it converts "it's almost always valid JSON" into "it always is". Even so, keep the defensive validation: the schema guarantees the shape, not the content (a confidence: "high" on a made-up answer is still an evaluation problem, not a format one), and your code will stay portable across providers.

Concept equivalence across providers

Everything above exists, under other names, in any serious provider. At the concept level (not concrete figures or identifiers):

Concept	Anthropic	OpenAI	Google Gemini
Python SDK	`anthropic`	`openai`	`google-genai`
API key	`ANTHROPIC_API_KEY` (env)	`OPENAI_API_KEY` (env)	`GOOGLE_API_KEY` (env)
System prompt	Separate `system` parameter	Message with `system`/`developer` role in the list	`system_instruction` in the configuration
Conversation turns	`messages` with `user`/`assistant` roles	`messages` with `user`/`assistant` roles	`contents` with `user`/`model` roles
Output cap	`max_tokens`	Output token limit parameter	`max_output_tokens`
Token usage	`usage.input_tokens` / `usage.output_tokens`	`usage` object (prompt / completion)	`usage_metadata`
Stop reason	`stop_reason`	`finish_reason`	`finish_reason`
Structured output	JSON schema in the request	`response_format` / strict schemas	`response_schema`

The lesson of this table: learn the concepts, not the names. If Nubelia switches providers tomorrow, 90% of your code (templates, validation, evaluation, error handling) does not change; only the thin layer that talks to the SDK. That is why we concentrate all API interaction in client.py — a single file to rewrite if that day ever comes.

What we have deliberately not touched today: the response arrives all at once after several seconds (streaming, 03-02), if the network fails the exception blows up uncontrolled (errors and retries, 03-02), every call starts from scratch remembering nothing (conversations, 03-03), and we don't know what this costs per month (costs, 03-04).

Common Mistakes and Tips

Hardcoding the API key "just to test". That "just to test" ends up in a commit. Environment variable from minute one, no exceptions.
Accessing response.content[0].text without checking the block type. It works today; it breaks the moment the model returns another block type. Always filter by block.type.
Ignoring stop_reason. The typical symptom: "the JSON parser fails randomly". The real cause: responses truncated by max_tokens. Check the stop reason before parsing.
Forgetting that max_tokens limits the output, not the input. The input is limited by the model's context window (01-04); they are different limits with different errors.
Reinventing the parser in every script. The 02-03 artifacts (clean_json, validate_contract, FALLBACK_RESPONSE) are importable modules. A single validation point, a single place to fix.
Tip: print usage in every test during development. In two days you will have a solid intuition of how many tokens each type of question consumes — and 03-04 will feel much more natural.
Tip: run the 02-04 evaluation set (or a 10-question sample) against the API as soon as ask_docubot() works. It is your first evaluation with real answers, and the comparison baseline for everything you optimize later.

Exercises

Exercise 1

Write a function call_diagnostics(response) that receives the API response object and returns a dictionary with: text (concatenated content), truncated (boolean: True if stop_reason == "max_tokens"), input_tokens, output_tokens and model. Test it with a real call and with max_tokens=20 to force truncation.

Exercise 2

Modify ask_docubot_raw() to accept an optional temperature parameter (default 0.2) and make 3 calls with the same question at temperature 0.0 and another 3 at 0.9. Compare the outputs: does what we studied in 02-04 hold, about needing several runs per case when comparing?

Exercise 3

The support classifier from 02-02 (classification_prompt() with CLASSIFICATION_EXAMPLES) has not touched the API yet. Write classify_query(text: str) -> str that sends it with max_tokens=20 and temperature 0.0, and returns one of the 6 categories (billing, permissions, integrations, api, incidents, other). If the model's output does not exactly match any category, return "other" (a conservative default, as in 02-03).

Solutions

Solution 1:

def call_diagnostics(response) -> dict:
    return {
        "text": "".join(
            b.text for b in response.content if b.type == "text"
        ),
        "truncated": response.stop_reason == "max_tokens",
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "model": response.model,
    }

With max_tokens=20 you will see truncated: True and text interrupted mid-sentence: exactly the case ask_docubot() intercepts before parsing.

Solution 2:

def ask_docubot_raw(question, documentation, temperature=0.2):
    return client.messages.create(
        model=MODEL,
        max_tokens=1024,
        temperature=temperature,
        system=SYSTEM_DOCUBOT,
        messages=[{"role": "user", "content": answer_prompt(question, documentation)}],
    )

for t in (0.0, 0.9):
    print(f"--- temperature={t} ---")
    for i in range(3):
        r = ask_docubot_raw(question, documentation, temperature=t)
        print(f"[{i+1}]", extract_text(r)[:120])

At 0.0 the three outputs will be almost identical (almost: the non-determinism from 01-04); at 0.9 they will vary visibly. Practical conclusion: comparing prompts with a single run per case is noise, not measurement — that is why 02-04 fixed 3 runs per case.

Solution 3:

from prompts import classification_prompt

VALID_CATEGORIES = {"billing", "permissions", "integrations", "api", "incidents", "other"}

def classify_query(text: str) -> str:
    response = client.messages.create(
        model=MODEL,
        max_tokens=20,
        temperature=0.0,
        messages=[{"role": "user", "content": classification_prompt(text)}],
    )
    output = extract_text(response).strip().lower()
    return output if output in VALID_CATEGORIES else "other"

Two decisions inherited from module 2: temperature 0.0 because classification demands consistency, and default "other" because when in doubt a generic category is better than a wrong one that looks confident.

Conclusion

DocuBot is no longer paper: you have a reproducible environment with the SDK installed and the key out of the code, a first real call sending the SYSTEM_DOCUBOT from 02-01 and the templates from prompts.py, and the judgment to read what comes back — the content as a list of blocks, stop_reason as a sentinel (especially max_tokens before parsing) and usage as the meter of the bill. The defensive parser from 02-03 now processes real responses, reinforced by native structured-output modes when the provider offers them, and the equivalence table guarantees that none of this ties you to a specific provider. But our integration is naive in one aspect the user notices instantly, and in another they will notice on the worst day of the year: the response takes several seconds to appear all at once, and if the API returns a 429 or a 500, DocuBot simply blows up. In the next lesson we solve both: streaming so the words flow as they are generated, and a serious strategy of errors and retries so DocuBot degrades gracefully instead of breaking.

First Integration with an LLM API

Contents

From paper to the API: what we are going to build

Setting up the environment: venv, SDK and API key

The first call: system + user

Anatomy of the response

Main parameters

Hooking the JSON parser from 02-03 up to a real response

Concept equivalence across providers

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Generative AI and LLMs for Developers Course

Module 1: Generative AI Fundamentals

Module 2: Prompt Engineering

Module 3: Integrating LLMs into Applications

Module 4: RAG - Retrieval-Augmented Generation

Module 5: Function Calling and Agents

Module 6: Toward Production