SALES_FILE has been declared since the previous lesson pointing at data/sales.csv, a file that does not exist yet. Its moment has come, and with it the problem none of the tools we have seen so far can solve.

The Escena Viva catalog takes two kilobytes: readFile swallows it whole without breaking a sweat. A season's sales history is another matter. Three venues, two hundred sessions a year, thousands of tickets per session, each with its code, purchase channel and timestamp: hundreds of megabytes. And readFile has no way of reading that without putting it all in memory.

Streams are Node's answer to that problem, and they are far more than an optimization: they are the model Node uses to think about input and output. The HTTP requests and responses of Module 4 are streams; compression, cryptography and network connections are streams. And — this will sound familiar — streams are EventEmitter objects, so everything from lesson 02-05 applies here directly.

1. The problem: measuring what readFile costs
2. What a stream is and its four types
3. Streams are EventEmitter objects
4. Flowing mode and paused mode
5. createReadStream, createWriteStream and highWaterMark
6. Backpressure: what write() returns
7. pipe() and its weak spot
8. The Escena Viva data/sales.csv file
9. Reading line by line with readline
10. for await...of over a stream

You should not take this on faith. Let's measure it.

With an 800 MB sales.csv, the output is devastating:

And if the file were any bigger, readFile would not even get to print: it would fail with ERR_STRING_TOO_LONG — a V8 string has a limit of roughly 512 MB — or with JavaScript heap out of memory, which kills the whole process. The difference is two orders of magnitude, and the explanation is simple: readFile needs the complete file in memory at once, while the stream processes it in 64 KB chunks and discards each one as soon as it has used it. The stream's usage does not depend on the size of the file: that is the key property, constant memory.

A stream is a sequence of data processed in pieces, as they become available, instead of waiting to have them all. The water analogy is a good one: readFile is filling the whole bathtub before touching the water; a stream is opening the tap and working with whatever comes out.

Transform streams are the subject of the next lesson. Here we concentrate on the first two, which are the foundation of everything else.

This is not an analogy: Readable and Writable inherit from EventEmitter. on, once, emit, off and everything from lesson 02-05 works as is, including the golden rule that now takes on its true meaning.

Two warnings worth half a lesson. The first: end and finish are not the same. end belongs to the reading side ("the input is over") and finish to the writing side ("I have finished flushing"); mixing them up leads to closing a file before its data is on disk. The second: a stream's error event follows the EventEmitter rule you already know — an error with no listeners throws an uncaught exception and kills the process. A stream without an on('error') is a production outage waiting to happen, and the usual cause is not a broken disk: it is an ENOENT because the file was not there.

A Readable has two ways of delivering data, and this duality is the source of half the trouble people hit when starting with streams.

Do not mix them. Subscribing to data and calling read() on the same stream produces behavior that is hard to reason about: lost chunks, an end that never arrives, scrambled ordering. Pick one mode per stream and stick to it — although in practice the recommendation is to use neither directly: pipe, pipeline or for await...of solve the problem better. And a third detail about flowing mode: if you subscribe to data after an await, you can lose chunks, because the stream started flowing before your listener arrived. Always subscribe synchronously, right after creating the stream.

The highWaterMark is the size of the internal buffer: how many bytes the stream accumulates before deciding it has enough. Lowering it (16 KB) spends less memory per stream in exchange for more system calls and more events; raising it (1 MB) does the opposite. The default 64 KB is reasonable almost always: lowering it makes sense when you handle thousands of simultaneous streams — a server with many connections — and raising it when you process a few very large files. In object mode (which you will see in the next lesson) the highWaterMark counts objects, not bytes, and defaults to 16.

A warning about encoding: if you do not set it, chunks arrive as Buffer objects; and if you set it to 'utf8', the stream makes sure no multibyte character is split between two chunks — a real problem we will look at in depth in lesson 03-06 with StringDecoder.

Here is the concept that separates people who use streams from people who understand them. Imagine you read from a fast SSD and write to a slow disk or a network connection: data comes in at 500 MB/s and goes out at 50 MB/s. Where does the difference go? Into the write stream's internal buffer, which grows without stopping until memory runs out. In other words: you have swapped a readFile that consumed 800 MB for a stream that consumes... 800 MB.

Backpressure is the mechanism that prevents that, and it rests on a detail almost everybody ignores:

write() returns a boolean. And that boolean is not informational: it is an order.

That code works with small files and blows up with large ones: the writer's buffer grows indefinitely because nobody stops feeding it. It is the most common mistake when writing streams by hand, and the hardest to diagnose, because in development — small file, fast disk — it never shows up.

The full cycle is: data → write() returns false → pause() → the buffer empties and the writer emits drain → resume() → back to data. The result is that the slow consumer sets the pace of the fast producer, and memory stays bounded by the highWaterMark. This idea is not exclusive to files: it is the same one that governs TCP and the one that will keep a slow client from taking down your HTTP server in Module 4.

Writing the pause/resume/drain dance by hand in every pipe would be unbearable. pipe() does it for you:

That line is equivalent to the whole previous block: it connects both streams, manages backpressure and calls writer.end() when the reader finishes. They can be chained — a.pipe(b).pipe(c) — because pipe returns the destination stream.

But pipe has a serious and little-known flaw: it neither propagates errors nor cleans up what it leaves behind. If reader fails — an ENOENT, a disk with errors — the error never reaches writer, which stays open with its descriptor unreleased; and if nobody listens for error on reader, the process dies.

With a pipe of three or four stages the problem multiplies: you have to subscribe on('error') on every stream and manually close whichever are left hanging. It is tedious, it is easy to forget one, and that oversight is paid in leaked descriptors until the EMFILE.

That is why the modern recommendation is blunt: use pipe() to understand the concept and stream.pipeline to do the work. pipeline propagates errors, destroys every stream in the chain when something fails and has a promise-based version. It is the central subject of the next lesson.

We need data to work with. This is the format of the sales history:

The columns follow the project conventions: ticketCode with the format EV-<year>-<6 digits>, sessionId shaped as ses-NNN-M, saleDate as an ISO string with no zone, priceCents an integer and channel one of three values (web, box-office, app). We generate a file consistent with the seed: exactly 1811 sales, distributed according to each session's sold count.

The comment about backpressure is deliberate: with 1811 lines there is no problem, but the correct code for real volumes would have to respect the return of write() or, better, be built with pipeline and a generator, as we will do in the next lesson.

A CSV file is processed by lines, but a stream's chunks do not line up with lines: a 64 KB chunk cuts whatever line falls on its edge in half. In the lab of section 1 we solved it by hand with a leftover variable; the node:readline module does it properly and with no code of your own.

Three decisions worth mentioning. crlfDelay: Infinity keeps a file generated on Windows from producing lines ending in an invisible \r that ruins the last field. A corrupt line is logged and skipped, it does not abort the report: in a history of millions of records there is always garbage, and a report that dies on line 4,000,000 is worth nothing. And memory usage is constant: bySession holds one entry per session (seven), not per sale.

That for await (const line of reader) deserves an explanation, because it is the most readable way of consuming a stream and you already know it from lesson 02-05, when you saw events.on().

Every Readable is an async iterable. You can walk it directly:

Its advantages over on('data') are concrete: backpressure is automatic (the loop does not ask for the next chunk until it finishes the body) instead of manual; errors are caught with a normal try/catch around the loop instead of a separate on('error'); you can use await inside; and a break destroys the stream on its own, with no explicit destroy().

That await inside the loop is the decisive difference. If every CSV line requires a database query, with on('data') the queries would all fire at once and sink the server; with for await, reading stops by itself until the query finishes, so backpressure comes for free. The price is raw throughput: for await is somewhat slower because each iteration creates a promise. For a batch process that is irrelevant; on a hot path with heavy traffic, measure it.

• Not subscribing on('error'). An error with no listeners on an EventEmitter kills the process, and a stream is an EventEmitter. The definitive solution is pipeline.
• Ignoring the return of write(). It works in development and exhausts memory in production. If you write by hand, respect the drain.
• Confusing end with finish. end is the reader exhausted; finish is the writer flushed. Closing on the wrong event truncates data.
• Assuming a chunk is a line. It never is. Use readline or a Transform that splits.
• Mixing flowing and paused mode, or subscribing to data after an await. In the first case behavior is erratic; in the second you lose chunks.
• Accumulating every chunk in an array to join them at the end. That is readFile with extra steps: you have lost the stream's only advantage.
• Tip: if the file fits comfortably in memory and you only read it once, readFile is simpler and faster; streams are for the huge, the endless, or whatever arrives bit by bit. And when you doubt whether backpressure is happening, print writer.writableLength now and then: if it grows without stopping, you are ignoring it.

Extend src/reports/sales-by-session.js with a summarizeByChannel() function that walks data/sales.csv a single time and returns, per channel, the number of tickets, the total revenue and the peak hour (the hour of the day with the most sales). It must use readline, keep constant memory and check that the sum of tickets across all channels is 1811.

Write src/lab/copy-with-backpressure.js that copies a large file without using pipe, respecting the drain, and prints every second: megabytes copied, the writer's writableLength and the number of backpressure pauses. Then run a version that ignores the return of write() and compare memory usage with process.memoryUsage().rss.

Write src/lab/filter-sales.js that reads data/sales.csv, keeps only the sales whose sessions belong to a given venue (--venue="Sala Boveda", cross-referencing the catalog) and writes the result to data/sales-<venue>.csv preserving the header. Use readline to read and a createWriteStream to write, respecting backpressure.

Solution 1. The key is to accumulate on two levels during the same pass:

The 24-slot array per channel is the trick that keeps memory constant: sales are not stored, only the counter for each hour. A Map of full dates would grow with the file.

Solution 2. The core of the manual copy:

With backpressure respected, writableLength hovers around the highWaterMark and rss stays flat. Without it, writableLength grows without limit and rss climbs toward the size of the file. Notice as well the two crossed on('error') handlers needed to avoid leaking descriptors: that manual work is exactly what pipeline automates.

Solution 3. What matters is cross-referencing the catalog before starting to read the CSV, so it does not happen per line:

That await new Promise(... 'drain' ...) is backpressure translated into the world of async/await, and it is a pattern worth memorizing: inside a for await, waiting for the drain also stops the reading, because the loop does not ask for the next item until it finishes the body. A Set with the session ids avoids walking the catalog 1811 times.

You now know why streams exist and exactly what problem they solve: you have measured the difference between readFile's 1621 MB and a stream's 6 MB on the same file, and you understand that a stream's usage does not depend on the size of the input. You know the four types — Readable, Writable, Duplex and Transform — and the fundamental fact that they are EventEmitter objects, with data, end, error, close, finish and drain, without confusing end with finish and knowing that an error with no listeners takes the process down.

You tell flowing mode from paused mode and why they must not be mixed; you know what the highWaterMark is and what you gain and lose by moving it. And, above all, you understand backpressure: that write() returns a boolean which is an order, that ignoring it turns your stream into a readFile in disguise, and that the pause → drain → resume cycle is what lets the slow consumer set the pace. You know pipe() automates that dance but neither propagates errors nor cleans up descriptors, and that is why it is not the final tool. Escena Viva, meanwhile, already has its data/sales.csv with the seed's 1811 sales and a src/reports/sales-by-session.js that walks it line by line with readline, accumulating tickets, revenue and channels per session with constant memory. And you have seen the most readable way of consuming any stream: for await...of, which throws in backpressure for free and lets you await inside the loop.

In Transform Streams and pipeline we close the circle. You will learn to write your own streams with Transform (_transform and _flush), to assemble complete processing pipes — raw CSV to objects, filtering by venue, aggregating by session, writing the report — and to replace pipe() with stream.pipeline, which propagates errors, destroys the whole chain when something fails and has a promise-based version. You will also see Readable.from(), async generators as transformations, zlib compression slotted into the pipe and cancellation with AbortSignal.