The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Costs, Latency and Caching

Next ►
►

Everything built in this module consumes two resources that don't show up in the code: money and time. Every token of SYSTEM_DOCUBOT, of the classifier's few-shot examples and of the 03-03 history is billed on every call; and every call takes an amount of time the Nubelia employee perceives. In this lesson we put numbers on both: we will understand the per-token pricing model, write a cost-estimation function and DocuBot's monthly projection (refining that ~€400/month estimate we eyeballed in 01-04), review the levers to lower it, measure latency with the right metrics, and finish with the lever that attacks cost and latency at once: caching, both the provider's prompt caching and a response cache of our own.

Contents

  1. The pricing model: input and output tokens
  2. Estimating the cost per call in Python
  3. DocuBot's monthly projection (refining the 01-04 figure)
  4. Cost levers
  5. Latency: TTFT, total time and how to measure them
  6. The provider's prompt caching: the stable prefix
  7. Your own response cache: hashing, TTL and validity
  8. Budgets and alerts: design

The pricing model: input and output tokens

LLM APIs bill by tokens processed, with two different rates normally expressed as a price per million tokens:

  • Input tokens: everything you send — system prompt, few-shot examples, documentation, history, question.
  • Output tokens: everything the model generates. It usually costs 3 to 5 times more than input, because generating is computationally more expensive than reading.

The concrete figures change frequently and depend on the model and the provider, so in this course we will always work with illustrative variables — replace them with your provider's current rates the day you calculate for real:

# pricing.py — ILLUSTRATIVE values, not real rates.
# Check your provider's pricing page and update here.

INPUT_PRICE = 3.0        # € per million input tokens (mid-tier model)
OUTPUT_PRICE = 15.0      # € per million output tokens

INPUT_PRICE_MINI = 0.5   # € per million — small model (for the classifier)
OUTPUT_PRICE_MINI = 2.5

# Typical factors for the provider's prompt caching (see section 6):
CACHE_READ_FACTOR = 0.1     # reading from the cache ≈ 10% of the input price
CACHE_WRITE_FACTOR = 1.25   # writing to the cache ≈ 25% surcharge

Two structural consequences of this model we have already been intuiting:

  • Input dominates in DocuBot-style systems. Our calls send thousands of tokens (system + few-shot + documentation) to generate a few hundred. Optimizing the input pays more than shortening answers.
  • Everything repeated is re-paid. SYSTEM_DOCUBOT travels in full in every one of the month's thousands of calls. That repetition is exactly what prompt caching attacks.

Estimating the cost per call in Python

The usage object from 03-01 gives us the exact data; all that's left is to put a price on it:

# costs.py
from pricing import INPUT_PRICE, OUTPUT_PRICE

def estimate_cost(usage, input_price: float = INPUT_PRICE,
                  output_price: float = OUTPUT_PRICE) -> float:
    """Cost in euros of a call, from its usage object."""
    input_cost = usage.input_tokens / 1_000_000 * input_price
    output_cost = usage.output_tokens / 1_000_000 * output_price
    return input_cost + output_cost

# Usage after any call:
# response = client.messages.create(...)
# print(f"This call: €{estimate_cost(response.usage):.4f}")

To estimate before calling (for example, to budget a batch), providers offer a token-counting endpoint that tokenizes without generating (in the Anthropic SDK, client.messages.count_tokens(...), which returns input_tokens); failing that, the rough rule from 01-02 (1 token ≈ 0.75 words in Spanish/English) works for orders of magnitude.

Hook estimate_cost up to the accounting in the Conversation class from 03-03 (which already accumulates total_input_tokens and total_output_tokens) and you get cost per conversation with no extra effort.

DocuBot's monthly projection

In 01-04 we estimated "~€400/month" almost by eye, as an order of magnitude to decide whether the project made sense. Now we have real usage data and a concrete architecture (classifier + answer, the 02-02 chaining). Let's refine with numbers that are fictional but consistent with what we measured in the module:

Profile of one DocuBot query (tokens, measured/estimated):

Component Input Output
Classification call (few-shot from CLASSIFICATION_EXAMPLES) ~900 ~10
Answer call: SYSTEM_DOCUBOT (~600) + injected documentation (~2,000) + question (~100) + JSON template (~150) ~2,850 ~300

Fictional volume: 250 employees, ~1,200 queries per working day, 22 days/month → 26,400 queries/month.

# projection.py
from pricing import (INPUT_PRICE, OUTPUT_PRICE,
                     INPUT_PRICE_MINI, OUTPUT_PRICE_MINI)

def call_cost(input_toks: int, output_toks: int,
              input_price: float, output_price: float) -> float:
    return input_toks / 1e6 * input_price + output_toks / 1e6 * output_price

def monthly_projection(monthly_queries: int = 26_400,
                       classifier_on_mini: bool = False) -> float:
    """DocuBot's monthly projection in euros (illustrative figures)."""
    cls_input_price = INPUT_PRICE_MINI if classifier_on_mini else INPUT_PRICE
    cls_output_price = OUTPUT_PRICE_MINI if classifier_on_mini else OUTPUT_PRICE

    classification_cost = call_cost(900, 10, cls_input_price, cls_output_price)
    answer_cost = call_cost(2_850, 300, INPUT_PRICE, OUTPUT_PRICE)

    return monthly_queries * (classification_cost + answer_cost)

print(f"Base (everything mid-tier): €{monthly_projection():.0f}/month")
print(f"Classifier on the small model: €{monthly_projection(classifier_on_mini=True):.0f}/month")

With the illustrative rates: the classification call costs ~€0.0029 and the answer call ~€0.0130 — that is, ~€0.016 per query and ~€417/month. The napkin estimate from 01-04 (~€400) was, for once, reasonable — but now we know where every cent comes from, and that is what turns a figure into a lever: 82% of the cost is the answer call, and within it 78% is input. We now know where to squeeze.

Cost levers

Ordered by effort-to-impact ratio for DocuBot, all built from what the module taught:

  1. A smaller model wherever capacity is overkill. The classifier picks among 6 labels with few-shot; it doesn't need the answer model. Moving it to the small model (classifier_on_mini=True) lowers its cost from ~€0.0029 to ~€0.00047 per query: ~€62/month saved by changing one parameter. The general rule: every task in the pipeline goes to the cheapest model that passes your 02-04 evaluation — the rubric decides, not intuition.
  2. The provider's prompt caching (section 6): the stable prefix (SYSTEM_DOCUBOT + few-shot) is billed at ~10% on cache-hit calls.
  3. A tuned max_tokens and a concise format. The system prompt already asks for <150 words (02-01); max_tokens=1024 is a safety net, not an invitation. Output is billed at the expensive rate: every filler word avoided is direct savings.
  4. Trimming the repeated input: are all 6 few-shot examples necessary, or does the 02-04 evaluation come out the same with 4? Can the injected documentation be more selective? (That last question is literally module 4: RAG is, among other things, a cost lever — injecting 2 relevant fragments instead of 10 generic ones.)
  5. History management (03-03). In conversations, truncation or the sliding window turn the accumulated quadratic cost into a linear one. LLM summarization costs one call, but it pays for itself if it avoids resending thousands of tokens per turn.
  6. Your own response cache (section 7): the repeated query doesn't call the API at all — zero cost and zero latency.

Latency: TTFT, total time and how to measure them

The second currency. Two distinct metrics we already brushed against in 03-02, now with proper names:

  • TTFT (time to first token): time from sending the request until the first token arrives. It is what the user perceives as the bot "reacting". With streaming, it is THE experience metric.
  • Total time: until the last token. It governs batch-process throughput and timeouts.

What influences each:

Factor Affects TTFT Affects total In your hands
Input size (prompt + history) ✅ (it must all be processed before the first token) ✅ Yes: levers 4 and 5
Output length ❌ ✅ (token by token) Yes: max_tokens, concise format
Chosen model (size) ✅ ✅ Yes: lever 1
Provider load / time of day ✅ ✅ No: only measure (and the 03-02 retries)
Provider prompt caching ✅ (the cached prefix isn't reprocessed) ✅ Yes: section 6

Measurement with the streaming from 03-02 — exercise 1 of that lesson was exactly this; we consolidate it as a utility:

# latency.py
import time

def measure_call(stream_fn, *args, **kwargs) -> dict:
    """Runs a streaming function and returns TTFT and total time."""
    start = time.perf_counter()
    ttft = None
    chunks = []

    for text in stream_fn(*args, **kwargs):     # stream_fn: generator of chunks
        if ttft is None:
            ttft = time.perf_counter() - start
        chunks.append(text)

    return {
        "ttft_s": round(ttft, 2),
        "total_s": round(time.perf_counter() - start, 2),
        "text": "".join(chunks),
    }

Two measurement tips: (1) measure percentiles, not averages — API latency has a long tail, and the p95 (the worst case 1 in 20 users sees) tells the truth the average hides; (2) measure over the 02-04 evaluation set, which already represents the real distribution of questions — that way cost, quality and latency are evaluated over the same cases.

The provider's prompt caching: the stable prefix

Providers offer prompt caching: if the beginning of your request (the prefix) is byte-for-byte identical to that of a recent request, the provider reuses the processing already done and charges you a fraction of the price for that part (cache reads ≈ 10% of the input price, with a surcharge the first time it is written ≈ +25%; the cache expires after a few minutes without use).

The key word is prefix: the cache matches from the very start of the request, and any differing byte invalidates everything that comes after it. Hence the golden design rule: the stable stuff first, the variable stuff last.

Look at our answer call through that lens:

[system: SYSTEM_DOCUBOT]                 ← identical in ALL calls ✅
[few-shot / fixed template]              ← identical in all ✅
[injected documentation]                 ← varies per question ❌
[user question]                          ← always varies ❌

SYSTEM_DOCUBOT and the CLASSIFICATION_EXAMPLES are the perfect candidate: thousands of identical tokens repeated in every one of the 26,400 monthly calls. In the Anthropic SDK you mark the end of the cacheable prefix with cache_control; other providers apply it automatically to repeated prefixes — the concept is the same:

response = client.messages.create(
    model=MODEL,
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": SYSTEM_DOCUBOT,
        "cache_control": {"type": "ephemeral"},   # "cache up to here"
    }],
    messages=[{"role": "user", "content": answer_prompt(question, documentation)}],
)

# Verification: is it working?
print(response.usage.cache_creation_input_tokens)  # tokens written to the cache
print(response.usage.cache_read_input_tokens)      # tokens READ from the cache (the savings)

If cache_read_input_tokens is 0 call after call, something is breaking the prefix. The usual silent saboteurs:

  • A timestamp or variable value interpolated into the system prompt ("Today is {date}..."): every call has a different prefix. Variable content goes at the end of the user message.
  • A prefix that is too short: there is a cacheable minimum (on the order of a thousand tokens depending on the model); shorter prefixes aren't cached, with no error.
  • Mixed versions: if half the fleet sends PROMPT_VERSION 1.2 and the other half 1.1, each variant competes for its own cache. Yet another reason for the disciplined versioning of 02-04.

What it's worth in DocuBot: the ~600 system tokens (plus few-shot if they share the prefix) at 10% instead of 100%, over 26,400 calls/month, are on the order of €40–50/month with our illustrative rates — plus a free TTFT improvement, because the cached prefix isn't reprocessed. In module 4, when the injected documentation grows, the "stable first" ordering will be even more profitable.

Your own response cache: hashing, TTL and validity

Prompt caching makes the call cheaper; a response cache eliminates it. If twenty employees ask "how do I export to CSV?" in the same week, why generate the same answer twenty times?

The classic implementation: key = hash of the normalized question, value = the validated answer, with a TTL (time to live) after which the entry expires:

# response_cache.py
import hashlib
import time
import unicodedata

def normalize(text: str) -> str:
    """Reduces trivial variants to the same cache key."""
    text = text.lower().strip()
    # strip accents: "café" and "cafe" must produce the same key
    text = unicodedata.normalize("NFKD", text)
    text = "".join(c for c in text if not unicodedata.combining(c))
    return " ".join(text.split())      # repeated whitespace -> one space

class ResponseCache:
    def __init__(self, ttl_seconds: int = 24 * 3600):
        self.ttl = ttl_seconds
        self.data: dict[str, tuple[float, dict]] = {}   # key -> (timestamp, answer)

    def _key(self, question: str) -> str:
        return hashlib.sha256(normalize(question).encode("utf-8")).hexdigest()

    def get(self, question: str) -> "dict | None":
        entry = self.data.get(self._key(question))
        if entry is None:
            return None
        timestamp, answer = entry
        if time.time() - timestamp > self.ttl:
            del self.data[self._key(question)]           # expired
            return None
        return answer

    def save(self, question: str, answer: dict):
        self.data[self._key(question)] = (time.time(), answer)

Integration in front of the 03-02 pipeline:

cache = ResponseCache(ttl_seconds=24 * 3600)

def answer_with_cache(question: str, documentation: str) -> dict:
    if (cached := cache.get(question)) is not None:
        return cached                       # €0, ~0 ms
    result = answer(question, documentation)   # the full 03-02 pipeline
    if result["confidence"] != "low":       # don't fossilize doubtful answers
        cache.save(question, result)
    return result

When caching is valid — and when it isn't. This cache is only correct if three conditions hold:

  1. Same question ⇒ same desired answer. True for documentation questions ("how do I export?"); false for anything that depends on the user or their state ("what plan am I on?"). In DocuBot it is best to cache only queries with no session context — in fact, to integrate the cache before the 03-03 history: if the question arrives inside a conversation with context, it doesn't apply.
  2. The source hasn't changed. If Nubelia's documentation is updated, the cached answers become stale. The TTL is the passive defense (24h bounds the lag); the active one is invalidating the cache when documentation is published — a hook in the editorial process that runs cache.data.clear() or invalidates per section.
  3. The answer deserved to be kept. Hence the confidence filter: caching a fallback or a low-confidence answer multiplies a one-off failure across every user who inherits the entry.

Normalization has a limit: "how do I export to CSV?" and "what's the way to get a CSV out of the project?" are the same question to a human and different keys to the hash. The exact cache only captures literal repetitions (which in internal support are more frequent than they seem: people copy and paste). The version that groups by meaning uses embeddings — the technique that opens module 4 and that will hand us, as a bonus, the semantic cache.

As with the 03-03 sessions: the in-memory dict works for one process; with several, the cache lives in a shared store (Redis is the usual choice) — the pattern doesn't change.

Budgets and alerts: design

The last piece, at the design level (the fine instrumentation — dashboards, traces — belongs to 06-04). A system billed by usage needs three numbers agreed on before production:

  1. Monthly budget (e.g. €500): the figure management has approved. From it derives the daily one (~€23 per working day) — the early signal.
  2. Alert threshold (e.g. 80%): crossing it notifies the team. It doesn't cut the service; it buys time to investigate (a legitimate adoption spike? a runaway retry loop? someone pasting huge documents?).
  3. Hard ceiling (e.g. 150%): crossing it makes the system degrade — for example, serving FALLBACK_RESPONSE for new queries or restricting to the cache — instead of spending without limit. It hurts, but it is the difference between a bad day and a bad bill.

In DocuBot's design this comes down to little code and much discipline: a per-call cost counter (estimate_cost dumped to a log, alongside the 02-04 evaluation CSV we already know), aggregated by day and compared against the thresholds. Additionally, providers offer spend limits and alerts in their billing console: turn them on as an external safety net — your counter can have bugs; theirs is the one that bills. And one design metric more valuable than the total: cost per resolved query. If it drops month over month while quality (the 02-04 rubric) holds, you are genuinely optimizing and not simply spending less to give worse answers.

Common Mistakes and Tips

  • Budgeting only the output (or only "the answers"). In DocuBot the input is ~80% of the cost: system, few-shot, documentation and history weigh more than what is generated.
  • Copying rates from a tutorial (including this one). Prices change; the *_PRICE variables exist to be updated. Record the rate's date next to the number in your documentation.
  • Putting the date/time inside the system prompt and wondering why prompt caching never hits. Variable content breaks the prefix: always at the end, in the user message.
  • Caching responses with no TTL and no invalidation. The day Nubelia's documentation changes, DocuBot will repeat obsolete instructions with total self-assurance — a hallucination manufactured by your own infrastructure.
  • Caching the fallbacks too. A transient API error gets fossilized and served to everyone. Filter what deserves to enter the cache (confidence, contract validity).
  • Measuring latency with averages. The average hides the tail: report p50 and p95. A p50 of 1.2s with a p95 of 9s is a real problem that an average of 2s disguises.
  • Tip: write down the three figures for every optimization — cost, latency (p95) and quality (02-04 rubric) — before and after. A lever that saves 30% of cost but fails the evaluation isn't a lever: it's a cutback.
  • Tip: check usage.cache_read_input_tokens in your logs during the first week after enabling prompt caching. It is the cheapest way to discover a silent prefix saboteur.

Exercises

Exercise 1

Extend monthly_projection() with two parameters: cache_hit_rate: float (the fraction of queries served by your own response cache, which cost €0) and prompt_caching: bool (if True, the ~600 system tokens of the answer call are billed at 10%). Compute the combined scenario: classifier on mini + 20% cache hit rate + prompt caching. How much is left of the ~€417/month base?

Exercise 2

The normalization in ResponseCache decides which questions share a key. For each pair, state whether they share a key under the current implementation and whether they should: (a) "How do I export to CSV?" / "how do i export to csv"; (b) "How do I export to CSV?" / "How do I export to PDF?"; (c) "How do I export to CSV?" / "What's the way to get a CSV out of the project?"; (d) "What plan am I on?" / "what plan am i on?". Justify (d) with special care.

Exercise 3

Design the function log_call(usage, call_type: str) that feeds the budget control: it must accumulate the day's cost (use estimate_cost) in a file or simple structure, and return one of three signals: "ok", "warning" (over 80% of the daily budget) or "ceiling" (over 150%). State where in the 03-02 pipeline you would integrate it and what answer() should do with each signal.

Solutions

Solution 1:

def monthly_projection(monthly_queries=26_400, classifier_on_mini=False,
                       cache_hit_rate=0.0, prompt_caching=False):
    cls_input_price = INPUT_PRICE_MINI if classifier_on_mini else INPUT_PRICE
    cls_output_price = OUTPUT_PRICE_MINI if classifier_on_mini else OUTPUT_PRICE

    classification_cost = call_cost(900, 10, cls_input_price, cls_output_price)

    answer_input = 2_850
    if prompt_caching:
        # ~600 system tokens at 10% of the input price
        answer_input = 2_850 - 600
        system_extra = 600 * CACHE_READ_FACTOR / 1e6 * INPUT_PRICE * 1e6 / 1e6
        # more readable: the cached system cost computed separately
        system_cost = 600 / 1e6 * INPUT_PRICE * CACHE_READ_FACTOR
    else:
        system_cost = 0.0
        answer_input = 2_850

    answer_cost = call_cost(answer_input, 300, INPUT_PRICE, OUTPUT_PRICE) + system_cost

    api_queries = monthly_queries * (1 - cache_hit_rate)   # the rest: own cache, €0
    return api_queries * (classification_cost + answer_cost)

print(f"€{monthly_projection(classifier_on_mini=True, cache_hit_rate=0.2, prompt_caching=True):.0f}/month")

Result with the illustrative rates: ~€270/month — about 35% less than the ~€417 base, without touching quality (the classifier changes model only if it passes the 02-04 evaluation, the cache only serves already-validated answers, and prompt caching is quality-neutral). Teaching note: the calculation omits the cache-write surcharge (~25% on cache misses) for simplicity; include it if you want the precise figure.

Solution 2:

  • (a) They share a key (lowercasing and whitespace collapsing even them out; the "?" is also worth stripping in normalize() — a reasonable improvement to the implementation). Should they: ✅ coherent.
  • (b) They don't share one (they differ in "CSV"/"PDF") and they shouldn't: they are different questions with different answers. Normalization must never erase semantic content.
  • (c) They don't share one, but a human would say they should. This is the limit of the exact cache: capturing this equivalence requires comparing meanings, not bytes — semantic caching with embeddings, module 4.
  • (d) They share a key... and they shouldn't be in the cache at all. The answer depends on who is asking: caching it would serve one employee's plan to another — an information leak between users, the same failure as exercise 3 of 03-03 but manufactured by the cache. The defense sits before normalization: exclude from the cache every session/user-dependent query (for example, by caching only the history-free path, or by filtering on the classifier's category).

Solution 3:

import datetime, json, pathlib

DAILY_BUDGET = 23.0   # € (≈ €500/month over 22 working days)
SPEND_FILE = pathlib.Path("daily_spend.json")

def log_call(usage, call_type: str) -> str:
    today = datetime.date.today().isoformat()
    data = json.loads(SPEND_FILE.read_text()) if SPEND_FILE.exists() else {}
    data.setdefault(today, 0.0)
    data[today] += estimate_cost(usage)
    SPEND_FILE.write_text(json.dumps(data))

    spend = data[today]
    if spend >= DAILY_BUDGET * 1.5:
        return "ceiling"
    if spend >= DAILY_BUDGET * 0.8:
        return "warning"
    return "ok"

Integration: in answer() (03-02), right after each successful call — that's where usage exists. On "ok" nothing happens; on "warning" the team is notified (a warning-level log today; an alerting channel once the 06-04 observability exists) but service continues; on "ceiling" new queries switch to degraded mode (own cache + FALLBACK_RESPONSE for anything uncached) until a human reviews. Note the coherence with the whole course: the decision to cut off or keep spending is human; the system only prepares it and bounds it.

Conclusion

With this lesson, the integration module is complete — and DocuBot stops being merely a system that works to become a system we know how to operate: it knows the price of every token it sends and receives (estimate_cost, on top of the 03-01 usage), it has a monthly projection with a first and last name that confirms and refines the 01-04 figure (~€417/month base, ~€270 after applying the levers), it distinguishes TTFT from total time and measures them in percentiles over the 02-04 evaluation set, it exploits the provider's prompt caching thanks to a design decision that now proves prescient (the SYSTEM_DOCUBOT and the few-shot examples of 02-01/02-02 are a perfect stable prefix), it avoids entire calls with its own response cache (normalized hash, TTL, invalidation on documentation publishing) and it operates within a budget with an alert and a ceiling whose final word is human. Looking back, module 3 turned the paper artifacts of module 2 into a living system: the first real call with system/user roles (03-01), streaming and a serious strategy of errors, retries and degradation (03-02), conversations with manufactured memory and isolated sessions (03-03) and, today, the economics and physics of the whole. But the Achilles' heel remains intact: DocuBot answers well when we hand it the right documentation — and so far we have handed it over ourselves, by hand, in every example. Who finds, among hundreds of pages of Nubelia documentation, the two fragments that answer each question? That is exactly the problem retrieval-augmented generation solves. In module 4 we build that piece: starting with embeddings and semantic search (04-01) — the technique that compares meanings instead of words and that, as a bonus, will resolve the limitation of our exact cache —, moving on to vector databases, documentation ingestion and chunking, and culminating in the complete RAG pipeline that will turn DocuBot into what we promised in 01-01: an assistant that truly knows Nubelia.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved