We have spent the whole module surrounded by Buffer objects without looking at them head on. When we wrote fs.readFile(EVENTS_FILE, 'utf8') we asked for a translation; when we left it out — copying a poster, compressing the history with zlib — what traveled through the code was a pure Buffer. The chunks of a binary stream are Buffer objects. The response of an HTTP request in Module 4 will arrive as a Buffer. A hash in Module 8 is a Buffer.

This lesson opens the box: what exactly a Buffer is and why Node had to invent it, how to create one without leaving security holes, what each encoding means, how binary numbers are read and why the order of their bytes matters. You will also see the two classic mistakes of treating bytes as if they were characters: a subarray that shares memory with the original, and an emoji cut in half between two chunks of a stream. And we will apply it to Escena Viva: checking that the poster an organizer uploads really is a PNG — by looking at its first bytes, not its extension — and generating the payload of a ticket's QR code in base64url, signed and comparable in constant time.

1. What a Buffer is and why it exists
2. Creating buffers: from, alloc and the danger of allocUnsafe
3. Encodings and what each one is for
4. Binary numbers and byte order
5. Operations: subarray, concat, copy, compare, indexOf
6. Bytes versus characters: the deceptive length
7. StringDecoder: never split a character in two
8. Escena Viva: validating the poster by its magic numbers
9. Escena Viva: a ticket's QR in base64url
10. Comparing secrets in constant time
11. Buffer, TypedArray and ArrayBuffer

JavaScript was born in the browser to manipulate text and documents. Until 2011 it had no type capable of representing binary data: only UTF-16 strings, floating point numbers and objects. Node, on the other hand, was born to read files and talk over sockets — that is, to move bytes. It needed a type the language did not give it, so it created one: Buffer.

A Buffer is a fixed-length sequence of bytes: each position holds an integer between 0 and 255, and it can neither grow nor shrink — for a bigger one you create another and copy. Two traits set it apart from a normal array:

• It is a Uint8Array. Literally: Buffer.prototype inherits from Uint8Array.prototype. Everything that works on a Uint8Array works on a Buffer, plus the methods Node adds on top (toString, write, readUInt32BE…).
• Its memory lives outside the V8 heap. The Buffer object you manipulate is on the heap, but the bytes sit in separately reserved memory. That lets the operating system write straight into it without copying, and lets a process move hundreds of megabytes without pressuring the garbage collector. It is also the reason why in lesson 03-04 heapUsed barely moved while rss did grow.

Notice how the console prints it: <Buffer ...> followed by the bytes in hexadecimal. If you see that in your logs where you expected text, the cause is almost always the same: you forgot the encoding when reading.

Watch the third and fourth rows, which look interchangeable and are not: Buffer.from(otherBuffer) copies, while Buffer.from(arrayBuffer) shares memory, so modifying one would affect the other only in the second case.

The name allocUnsafe is not documentation hyperbole. Buffer.alloc(n) walks the n bytes setting them to zero; allocUnsafe(n) hands them over exactly as they sat in the reserved memory, which may contain leftovers of the same process's earlier data: fragments of a JSON file read a second ago, pieces of a password, half an HTTP header.

In 2018 this difference produced a whole family of information leaks in npm packages that reserved a buffer with allocUnsafe and sent it over the network without filling it completely: the spare bytes traveled with whatever had been there before. The practical rule is simple: always use Buffer.alloc; allocUnsafe only when the next line will overwrite the entire buffer and the performance gain has been measured and justified.

An encoding is a translation contract between bytes and characters. Node supports these:

Two important clarifications:

• base64 is not encryption. It is a reversible representation with no secret whatsoever. It exists to put bytes where only printable characters fit, not to protect anything.
• base64url is the URL-safe variant: it replaces + with -, / with _ and drops the = padding. It is exactly what the JWTs of Module 8 use and what we will use for the ticket QR codes, because a + inside a URL is read as a space and a / splits the path.
• An unknown encoding does not fail silently: Buffer.from(x, 'utf-9') throws TypeError: Unknown encoding, and you can check the list with Buffer.isEncoding('base64url').

A binary format does not store "1189" as text: it stores the number in a fixed number of bytes. Reading it requires knowing how many bytes it takes, whether it is signed and in what order its bytes sit. That last point is the byte order, or endianness:

• BE (big endian): the most significant byte first. It is the order of networks and of most file formats (PNG, JPEG).
• LE (little endian): the least significant first. It is the native order of the usual x86 and ARM processors.

The same four bytes are worth 1189 or 2768994304 depending on how you read them. That is why the order is not guessed: the format fixes it and you have to read its specification. The methods all follow the same pattern: read/write + U if unsigned + Int + size in bits + BE/LE. For 64 bits there are readBigUInt64BE and friends, which return a BigInt. Writing an out-of-range value or at an offset beyond the buffer throws ERR_OUT_OF_RANGE: it is a loud error, not silent corruption.

The classic mistake is in the first row. On an array, slice returns a copy; on a Buffer, slice and subarray return a window onto the same memory:

If you need a real copy, ask for it explicitly: Buffer.from(original.subarray(3, 7)) or Buffer.copyBytesFrom(...). This detail is the cause of a very hard bug to find: storing chunks of a stream with chunks.push(data) without copying them, when the stream reuses the same internal buffer between reads; the accumulated chunks all end up holding the same thing, the last one. As for concat, it is the correct way of rebuilding accumulated content — Buffer.concat(chunks) after a for await over a stream — and it accepts a third argument with the total length, which saves recomputing it if you already know it. That pattern is exactly what readFile does internally, with the same memory consequence we measured in 03-04: use it only when you know the content is small.

String.length counts UTF-16 units. Buffer.length counts bytes. And Buffer.byteLength(string) tells you how many bytes a text will take before converting it. The three numbers can all differ:

Three lessons from this table. First: an ó takes two bytes in UTF-8, so reserving a buffer with Buffer.alloc(text.length) to hold accented text truncates it. Second: an emoji takes two units in String.length and one when iterated with [...text], because JavaScript represents characters outside the basic plane as a pair of surrogate values. Third, and the one that really breaks programs: cutting a buffer at an arbitrary position can split a character in two.

Converting each chunk to text separately destroys the split character: the � (U+FFFD) can no longer be undone. Joining the bytes first and decoding afterwards works, but it requires having all the content in memory, which is exactly what a stream avoids.

The previous problem is not theoretical: it is exactly what happens when a stream delivers 64 KB chunks and character number 65,536 falls astride two of them. The solution is in node:string_decoder, a stateful decoder that holds back the incomplete bytes at the end of a chunk until the missing ones arrive.

The first write returns "Sala B" without the ó: the decoder saw the character's first byte and kept it. The second write completes it and emits óveda. The final end() returns whatever was left pending — if the flow ends with an incomplete character, that is where the � appears, a sign that the input was truncated. When you pass { encoding: 'utf8' } to createReadStream or call setEncoding('utf8') on a stream, Node uses a StringDecoder internally, and that is why the chunks already arrive as correct text. You only need to use it by hand when working with raw Buffer objects: when decrypting, when decompressing chunk by chunk or when implementing a protocol of your own.

Organizers upload the poster for their event. The file extension proves nothing: renaming virus.exe to poster.png takes a second. What does identify a format is its first bytes, the so-called signature or magic number.

We put the FileHandle API of lesson 03-02 to work to read only the first bytes, instead of loading a ten-megabyte image to look at eight:

Four decisions worth commenting on: Buffer.alloc and not allocUnsafe, because if the file is under 16 bytes the spare ones would be memory garbage compared against real signatures; Buffer.from(target.subarray(...)), which returns an independent copy instead of a view with access to the spare bytes; equals instead of toString, because comparing bytes with bytes removes any encoding doubt — 0x89 is not valid UTF-8 text; and error.appCode following the course's domain convention, with the header in hex attached to diagnose what the organizer actually uploaded.

The check works. Mind the scope of this technique: the signature proves the format, not that the content is harmless. In Module 4, when receiving real uploads, we will combine it with a size limit and with the hardened path of lesson 03-03.

Every Escena Viva ticket carries a code such as EV-2026-000123 and a QR the usher scans at the door. The QR holds a URL, and inside that URL travels a payload: the ticket data plus a signature that prevents forgery. That content is binary and has to fit in a URL with no escaping: the exact use case for base64url.

areEqual is the safe comparison function of the next section, which lives in this same module. The body is not encrypted — anyone can decode it with Buffer.from(body, 'base64url').toString() — and that is acceptable: there is nothing secret in a ticket's QR. What the signature protects is integrity: without knowing the secret nobody can forge a valid ticket or change its session. Comparing that signature, however, cannot be done with ===, and that is the last technical section of the lesson.

a === b on strings compares character by character and stops at the first one that differs. That optimization, harmless in any other context, leaks information when what is compared is a secret: an attacker measuring response time can deduce how many leading characters they got right and rebuild the signature byte by byte.

crypto.timingSafeEqual always compares every byte, however long it takes to find the first difference. It has one requirement: both buffers must be exactly the same length, or it throws ERR_CRYPTO_TIMING_SAFE_EQUAL_LENGTH, and you do not control the length of someone else's data. The standard solution is to normalize the length with a hash before comparing:

digest() with no argument returns a Buffer — with 'hex' it would return a string — and two SHA-256 values are 32 bytes each no matter what. We will pick this function up untouched in Module 8, where the same reasoning applies to API keys and session tokens.

The three pieces fit together like this: an ArrayBuffer is a raw block of memory that cannot be read or written directly; a TypedArray (Uint8Array, Int16Array, Float64Array…) is a view that interprets that block as numbers of a certain type, and several views can look at the same block; a Buffer is a Uint8Array with methods added by Node.

There lies the subtlest trap in the module. Node allocates small buffers (under 4 KB) inside a shared 8 KB pool, so buf.buffer is not your buffer's memory: it is the whole pool's, and buf.byteOffset says where your slice starts. Passing buf.buffer to an API expecting the exact data hands over 8 KB of somebody else's memory. The correct way to convert:

In practice this shows up when interoperating with web standard APIs inside Node — fetch, crypto.subtle, WebSocket, the worker threads of Module 10 — which speak Uint8Array and ArrayBuffer, not Buffer.

• Seeing <Buffer 7b 0a ...> where you expected text. The encoding is missing: readFile(file, 'utf8') or .toString('utf8'). Never use String(buf) hoping it will do the right thing with binary data.
• Using allocUnsafe out of habit. If you do not fill the whole buffer, you publish the process's old memory. alloc by default, always.
• Believing slice copies. It shares memory, the opposite of arrays. Copy explicitly with Buffer.from(view) if the original is going to change or be reused.
• Allocating with text.length instead of Buffer.byteLength(text). A single accent already leaves you a byte short and the text comes out truncated.
• Concatenating chunks as strings. A chunk.toString() per stream chunk splits multibyte characters: accumulate Buffer objects and decode at the end, or use StringDecoder.
• Comparing signatures with ===. It leaks information through timing. timingSafeEqual over buffers of equal length, normalized with a hash.
• Tip: work with Buffer objects until the last possible moment and convert to text once, at your system's boundary. Every round trip costs CPU and is an opportunity to corrupt data.

Write src/lab/poster-inventory.js that walks a directory (by default data/posters/) and, for each file, prints in a table its name, its declared extension, the real type detected with detectType and whether they match. It must clearly flag the files whose extension lies. Use readdir with withFileTypes from lesson 03-02 and the paths from src/config/paths.js.

Write a function splitText(text, maxBytes) that returns an array of Buffer objects of at most maxBytes bytes each without splitting any character. Verify with 'Noche de Monólogos 🎭 en la Sala Bóveda' and maxBytes = 10 that concatenating the chunks and decoding gives back the original text and that no chunk produces a � on its own.

Design a minimal binary format for the daily occupancy report: 4 bytes of signature 'EVIV', 1 version byte, 2 big endian bytes with the number of sessions, 4 big endian bytes with the tickets sold and the rest as UTF-8 JSON. Write writeBinaryReport(file, report) and readBinaryReport(file), and check that the full round trip returns the original data and that a file with the wrong signature throws an error with appCode: 'UNSUPPORTED_FORMAT'.

Solution 1. The comparison between declared extension and real type is the core:

The .jpeg/.jpg case is a reminder that a format can have several legitimate extensions: the check must not raise false alarms. An unknown is not always an attack — it may be a format missing from SIGNATURES — but it is reason enough not to accept it.

Solution 2. The key is not to cut blindly, but to step back to the start of a character. In UTF-8, continuation bytes have the shape 10xxxxxx, that is, (byte & 0xc0) === 0x80:

The Buffer.from(...) around the subarray is essential: without it, the chunks would be views onto the same buffer and any later modification would affect them all. A shorter alternative if you only want text: walk [...text] accumulating by the Buffer.byteLength of each character.

Solution 3. The format is written and read in the same order in which it is defined:

With the seed data, readUInt32BE(7) returns 1811 and readUInt16BE(5) returns 7. Notice the version byte: it is what will let you read old files when the format changes, and leaving it out is the most frequent mistake when designing a binary format of your own. A writeUInt16BE with more than 65,535 sessions would throw ERR_OUT_OF_RANGE, a reminder that every binary field has a ceiling that must be chosen deliberately.

You now know what is inside the box. A Buffer is a Uint8Array over memory outside the V8 heap, of fixed length, which exists because JavaScript had no binary type when Node needed one. You create it with Buffer.from — copying from text, an array or another buffer, sharing from an ArrayBuffer — or with Buffer.alloc, never with allocUnsafe unless you are going to overwrite it entirely, because its initial content is memory the process itself has used.

You know the encodings and how the roles are shared out: utf8 for text, latin1 to treat bytes as characters one to one, hex for debugging and hashes, base64 to put binary inside JSON and base64url to put it inside a URL. You can read and write integers with readUInt32BE and friends, and you know the byte order is fixed by the format, not by your machine. You have mastered the operations and their traps: subarray shares memory — the classic mistake —, concat copies, equals compares with no encoding ambiguity. And you are clear on why String.length, Buffer.byteLength and [...text].length give three different numbers, why cutting a buffer at an arbitrary position produces an unrecoverable � and how StringDecoder avoids it by holding the incomplete bytes between one chunk and the next.

Escena Viva takes away two new modules: src/utils/file-type.js, which validates an event's poster by its magic number reading only sixteen bytes with FileHandle, and src/utils/ticket-qr.js, which encodes the QR payload in base64url with an HMAC signature compared in constant time with timingSafeEqual over equal-length digests. And you know that buf.buffer is not your buffer but the 8 KB pool it lives in: byteOffset and byteLength are not optional. With this you close Module 3. Escena Viva has gone from having its data inlined in the code to reading its catalog from disk asynchronously, organizing reports by month, resolving paths that work from any directory and on any system, processing a sales history with constant memory through pipeline pipes, and understanding the bytes flowing through all of the above. The project can read and write; what it still cannot do is talk to anyone. That starts in Module 4: HTTP, and you will see right away that it is not new territory: a Node HTTP server is an EventEmitter that emits request; the request you receive is a readable stream and the response you send back is a writable stream; the JSON body of a POST arrives in Buffer chunks that must be accumulated carefully; and serving a static file is exactly createReadStream plus the hardened path of lesson 03-03. Everything from this module comes back, this time wired to the network.