Throughout the module we have treated events as discrete units: an order is created, a payment is confirmed. But there is a class of problems in which events arrive continuously and at high speed —user clicks, IoT sensor readings, card transactions, log lines— and the value lies in processing them as they flow, without waiting to accumulate them. This is real-time data streaming: treating data as an infinite (unbounded) flow over which we apply transformations, aggregations, and pattern detection within milliseconds.

Streaming is the foundation of use cases such as instant fraud detection, live dashboards, real-time recommendations, or systems monitoring. In this lesson you will understand the fundamental difference between batch processing and stream processing, the role of temporal windows, and how Kafka Streams lets us implement all of this with relatively simple code.

1. Batch vs Streaming: the fundamental difference
2. Stream processing concepts
3. Windows: grouping time
4. Kafka Streams: processing on top of Kafka
5. Example: window-based fraud detection
6. Real-world use cases
7. Common mistakes and tips
8. Exercises and solutions
9. Conclusion

Batch processing operates on a finite and complete dataset (for example, "all of yesterday's sales") at a scheduled moment. Stream processing operates on an infinite flow of events as they arrive.

The key phrase: in batch you have all the data before starting; in streaming you never have all of it, because one more can always arrive. This forces you to reason differently, especially about time.

Before programming, let's set the vocabulary:

• Stream: an infinite sequence of events ordered (approximately) in time.
• Stateless transformations: they process each event independently. Examples: filter (discard), map (transform).
• Stateful transformations: they need to remember information from previous events. Examples: count, sum, aggregate by key. They require a state store.
• Event time vs processing time:
  • Event time: when the event occurred (timestamp in the message itself).
  • Processing time: when the system processes it.
  • They can differ greatly (an event from a phone without coverage can arrive minutes late). Serious processing is based on event time.
• Late data: events that arrive later than expected. The system must decide whether to incorporate or discard them.

Since the flow is infinite, we cannot "sum everything". Instead, we group the events into temporal windows and compute over each window. There are three main types:

Example: to detect fraud we look for "more than 3 payments in 1 minute from the same card". That is a stateful aggregation over a 1-minute window grouped by card.

Kafka Streams is a Java library that consumes from Kafka topics, processes the data (filtering, aggregation, windows, joins), and writes the results to other topics. It does not need a separate cluster: it is code you deploy as a normal application and that scales by adding instances.

Concepts of its API:

• KStream: represents an event flow (each record is an independent fact).
• KTable: represents a state table (each key has its latest value; new records update it).
• State store: a local store (backed in Kafka) where stateful aggregations are kept, fault-tolerant.

• application.id is critical: it identifies the application and names its internal topics and state stores. Two instances with the same application.id form a group and share the work.
• processing.guarantee: exactly_once_v2 enables exactly-once processing within Kafka (between topics), relying on transactions.

Let's detect cards with more than 3 payments within a 1-minute window:

Step-by-step explanation:

1. builder.stream("payments") creates a KStream that reads each payment from the input topic.
2. groupBy re-groups by cardId: now the events of the same card share a key and are processed together. It is the basis of any stateful aggregation.
3. windowedBy(TimeWindows.ofSize... 1 minute) defines tumbling windows of one minute. withNoGrace means we do not wait for late data.
4. count(...) keeps, in a fault-tolerant state store, how many payments each card has in the current window.
5. toStream() converts the KTable of counts into a change stream so we can keep processing.
6. filter(... count > 3) lets through only the card-window combinations that exceed the fraud threshold.
7. map builds a FraudAlert and to("fraud-alerts") publishes it to the output topic, where another service can block the card or notify the customer.

The result is a fraud detector that reacts in real time, without nightly batches.

• Fraud detection: anomalous transaction patterns instantly (like the example).
• Monitoring and alerting: detect service outages or error spikes in live logs.
• Real-time dashboards: sales per minute, users active right now.
• Recommendations: adjust suggestions based on the user's current browsing.
• IoT and telemetry: aggregate readings from thousands of sensors and detect thresholds (temperature, vibration).
• Event enrichment: join a stream of orders with a customer table to add data on the fly.

• Confusing event time with processing time. If you aggregate by processing time, a delayed event falls into the wrong window and the results are incorrect. Base the windows on event time whenever the order matters.
• Ignoring late data. Explicitly decide your policy: grace period, discard, or reprocess. Not deciding is deciding badly.
• Forgetting that state grows. Stateful aggregations consume memory/disk. Use windows with bounded retention and clean up stale state.
• Expecting "exactly-once" end to end. Kafka Streams guarantees it between Kafka topics, but when writing to external systems you again need idempotency (lesson 05-02).
• Using streaming when a batch would have sufficed. If a latency of hours is acceptable, a batch is simpler and cheaper. Streaming adds operational complexity.
• Tip: start with stateless transformations (filter, map); introduce state and windows only when the use case requires it.

1. For each case, indicate whether you would use batch or streaming and why: (a) monthly closing accounting report; (b) alert when a sensor exceeds 80 °C; (c) nightly calculation of commissions; (d) counter of currently connected users.
2. You want to compute a "moving average of temperature over the last 10 minutes, updated every 2 minutes". What type of window would you use and why?
3. Explain in your own words the difference between KStream and KTable and give an example of data suitable for each one.

1. (a) Batch: finite data for the month, no urgency. (b) Streaming: requires an immediate reaction. (c) Batch: scheduled nightly process over complete data. (d) Streaming: live metric that changes continuously.
2. A hopping/sliding window of size 10 minutes with a hop of 2 minutes. It is sliding because the windows overlap: every 2 minutes a new result is emitted that covers the last 10 minutes, which is exactly what a moving average requires.
3. A KStream models a flow of independent facts (each record is an event that adds to the previous ones), for example each payment or each click. A KTable models a state per key where each new record updates the previous value of that key, for example the current balance of each account or the latest price of each product.

Data streaming treats events as an infinite flow and processes them as they arrive, achieving latencies of milliseconds versus the hours of batch. We learned to reason about time (event time vs processing time), to group events into windows (tumbling, sliding, and session), and to implement stateful aggregations in Kafka Streams through a real fraud detector. We also saw when streaming is worthwhile and when a simpler batch is enough.

With this lesson we close Module 5: Event-Driven Architectures and Messaging. You have traveled the complete path: from the fundamentals of events and their asynchronous messaging, through storage with Event Sourcing and CQRS, the coordination of distributed transactions with the Saga pattern, to real-time stream processing. These patterns constitute the backbone of modern distributed systems and prepare you to design scalable, resilient, and reactive architectures.