The Project | About Us | Contribute | Donations | License

HOME

We have spent two modules carrying an outstanding debt. data/events.json has existed since the very first lesson, it holds the three events and seven sessions of Escena Viva, and the application has never read it: src/catalog-data.js still returns an array hardcoded into the code itself. Today we settle that debt.

The fs (file system) module is Node's door to the disk. It is also the place where everything from Module 2 stops being theory: fs.promises are the promises you already know, readFileSync is exactly the kind of operation that freezes the event loop, and choosing between the two is the difference between a server that answers a thousand users and one that goes silent while it reads a file.

By the end you will have turned getCatalog() into an asynchronous function that reads JSON from disk, you will understand why asynchrony propagates upward, you will tell file system errors apart by their code, and you will be able to write a file with no risk of leaving it half-written if the process dies mid-operation.

The three APIs of the fs module
readFile, encodings and what happens if you omit them
When the synchronous version is acceptable (and when it is a disaster)
The central task: an asynchronous data layer
Asynchrony propagates upward
File system errors and their codes
Why checking first with exists is a bad idea
Writing: writeFile, appendFile and the flag option
Atomic writes: temporary file plus rename
Memoizing the catalog so we do not re-read it

The three APIs of the `fs` module

Node offers three different ways of doing the same thing with files. This is not accidental historical redundancy: each one has its moment.

// 1. Promises (the one we will use throughout the course)
const content = await require('node:fs/promises').readFile('data/events.json', 'utf8');

// 2. Error-first callbacks (Node's original API)
require('node:fs').readFile('data/events.json', 'utf8', (error, content) => { /* ... */ });

// 3. Synchronous (blocks the main thread)
const content = require('node:fs').readFileSync('data/events.json', 'utf8');

API	How you import it	Does it block the loop?	Error handling	When to use it
Promises	`require('node:fs/promises')`	No	`try/catch` with `await`	By default, always
Callbacks	`require('node:fs')`	No	First argument of the callback	Legacy code; APIs that demand a callback
Synchronous	`require('node:fs')`, `Sync` suffix	Yes	Plain `try/catch`	Only at startup and in one-shot scripts

Two details: node:fs/promises and node:fs are different modules (you can also reach the first one through require('node:fs').promises, but importing the submodule is more explicit), and the functions with the Sync suffix only exist in node:fs: the promise API has none of them, by design.

`readFile`, encodings and what happens if you omit them

readFile opens the file, reads it whole into memory and closes it. That word — whole — is its defining feature and its limit: an 800 MB file takes 800 MB of memory. That is what streams are for, in lesson 03-04; for a JSON file of a few kilobytes, readFile is the right tool.

const withEncoding = await fs.readFile('data/events.json', 'utf8');
console.log(typeof withEncoding);                 // 'string'

const withoutEncoding = await fs.readFile('data/events.json');
console.log(Buffer.isBuffer(withoutEncoding));    // true
console.log(withoutEncoding.slice(0, 12));
// <Buffer 5b 0a 20 20 7b 0a 20 20 20 20 22 69>

Without an encoding, readFile returns a Buffer: a sequence of raw bytes. With an encoding, Node decodes those bytes into text and hands you a string.

The reason is that a file does not contain text: it contains bytes. Whether those bytes mean "Concierto" or the pixels of a poster depends on how you interpret them. 'utf8' tells Node: these bytes are UTF-8 encoded text, convert them. Buffer objects are the entire subject of lesson 03-06; for now the rule is enough: 'utf8' for JSON, CSV, configuration or any text; no encoding for images, PDFs, ZIPs or any binary.

A warning about the path: 'data/events.json' is relative to the directory you launched the process from, not to the file containing that line. It is one of the most common sources of error in Node and we will take it apart in lesson 03-03; for now, always run from the project root.

When the synchronous version is acceptable (and when it is a disaster)

readFileSync is not "the easy version". It is an operation that stops the main thread completely: during that time the event loop does not advance, no request is served and no timer fires. Let's measure it with the monitor from Module 2.

// src/lab/compare-blocking.js
// Demonstrates the effect of readFileSync on the event loop.

const fs = require('node:fs');
const fsPromises = require('node:fs/promises');
const { startMonitoring } = require('../utils/loop-monitor.js');

const FILE = 'data/events.json';
const REPETITIONS = 400;

async function measure(label, task) {
  const start = process.hrtime.bigint();
  await task();
  console.log(`${label}: ${(Number(process.hrtime.bigint() - start) / 1e6).toFixed(1)} ms`);
}

async function main() {
  // The monitor warns on stderr every time the loop lags more than 20 ms.
  const stop = startMonitoring({ intervalMs: 20, thresholdMs: 20 });

  await measure('sync ', () => {
    for (let i = 0; i < REPETITIONS; i += 1) fs.readFileSync(FILE, 'utf8');
  });
  await measure('async', async () => {
    for (let i = 0; i < REPETITIONS; i += 1) await fsPromises.readFile(FILE, 'utf8');
  });

  stop();
}

if (require.main === module) main();

node src/lab/compare-blocking.js
# [loop] lag of 31.4 ms
# sync : 38.2 ms
# async: 61.7 ms

Both results are surprising and both matter. The synchronous version is faster overall: there are no callbacks to schedule, no trip through the thread pool, no return to the event loop. And at the same time it blocked the loop for 31 ms, while the asynchronous version never blocked it once; during those 31 ms your server was dead to everyone. That is the key many people never quite absorb: asynchrony is not faster, it is fairer. It does not optimize one user's work, it lets you serve everyone else while that work happens. With a single user, Sync wins; with five hundred, Sync is a catastrophe.

Context	Is `Sync` acceptable?	Reason
Process startup, before listening for requests	Yes	Nobody is waiting yet; 30 ms of startup hurt no one
One-shot command line script	Yes	There is no concurrency to protect
Inside an HTTP request	Never	It freezes every connected user, not just the one asking
Inside a loop over many files	Never	The blocking is multiplied by the number of files

Operating rule: if the process is already serving traffic, Sync is forbidden.

The central task: an asynchronous data layer

The moment has come. src/catalog-data.js was written in Module 2 with the data inlined and with a signature designed precisely for this change. We replace it entirely:

// src/catalog-data.js
// Data access layer for the Escena Viva catalog.
// Reads the data/events.json seed and returns domain objects.

const fs = require('node:fs/promises');

const { Event } = require('./domain');

const EVENTS_FILE = 'data/events.json';

// Builds a domain error while keeping the original cause.
function fail(code, message, cause) {
  const error = new Error(message);
  error.appCode = code;
  error.cause = cause;
  return error;
}

// Reads the seed file and returns the parsed plain data.
async function readEventsFile() {
  let content;

  try {
    content = await fs.readFile(EVENTS_FILE, 'utf8');
  } catch (error) {
    if (error.code !== 'ENOENT') throw error;
    throw fail('DATA_UNAVAILABLE', `${EVENTS_FILE} not found`, error);
  }

  try {
    return JSON.parse(content);
  } catch (error) {
    // JSON.parse throws SyntaxError: we translate it into domain vocabulary.
    throw fail('CORRUPT_DATA', `${EVENTS_FILE} does not contain valid JSON`, error);
  }
}

// Returns the whole catalog as Event instances.
async function getCatalog() {
  const data = await readEventsFile();
  return data.map((record) => Event.fromJSON(record));
}

// Returns a single event by its id, or undefined if it does not exist.
async function getEventById(id) {
  return (await getCatalog()).find((event) => event.id === id);
}

module.exports = { getCatalog, getEventById, EVENTS_FILE };

Four decisions worth pointing out. structuredClone is no longer needed: we used to copy because the array lived in the module cache and the whole process shared it; now every call reads the file and builds fresh instances, so isolation comes for free. The layer returns domain objects, not plain data: Event.fromJSON has been waiting for this moment since Module 1, and whoever consumes the catalog receives Event objects with their getters, not anonymous dictionaries. Errors are translated into domain vocabulary through error.appCode, because the caller should not have to know about ENOENT or SyntaxError. And cause preserves the original error: it is standard since Node 16 and console.error prints it automatically, so you keep the trace while still giving a readable message.

Asynchrony propagates upward

Now getCatalog() returns a promise. That change does not stay inside the module: it climbs the whole call chain, and src/catalog.js has to adapt.

// src/catalog.js (excerpt: only main changes)

async function main() {
  const options = readOptions(process.argv.slice(2));

  let events;
  try {
    // The only real change: an await. The conversion .map is gone,
    // because the data layer already returns Event instances.
    events = await getCatalog();
  } catch (error) {
    console.error(`[catalog] ${error.message}`);
    if (error.cause) console.error(`[catalog] cause: ${error.cause.message}`);
    process.exitCode = 1;
    return;
  }

  // ...the rest (filters, showSummaryTable, showCatalog, totals) stays the same.
}

if (require.main === module) {
  // main() now returns a promise: its rejection has to be caught.
  main().catch((error) => {
    console.error('[catalog] unexpected error:', error);
    process.exitCode = 1;
  });
}

Asynchrony is contagious upward: if a function awaits something asynchronous, it becomes asynchronous itself, and so does its caller. The chain readFile → readEventsFile → getCatalog → main ends at a point called the boundary, where somebody has to decide what to do with the error. In a console script that boundary is the require.main === module block; in Module 4 it will be the HTTP request handler. What must never happen is calling main() without a .catch(): that would be an unhandled promise rejection and, as you saw in Module 2, it brings the process down with an unhandledRejection.

File system errors and their codes

fs errors carry a code property with a stable identifier. Never compare the text message: it changes between versions and system languages.

`code`	Meaning	Usual cause
`ENOENT`	No such file or directory	Misspelled path, deleted file, missing parent directory
`EACCES`	Permission denied	The process user has no read or write permission
`EISDIR` / `ENOTDIR`	It is a directory / a segment is not	Reading a folder as a file; `events.json/other.txt`
`EEXIST`	The file already exists	Writing with `flag: 'wx'`
`EMFILE`	Too many open files	Unclosed descriptors; opening thousands at once
`ENOSPC`	No space left on the disk	Full disk while writing
`EPERM`	Operation not permitted	Locked file (typical on Windows)

The error also carries error.path (the path involved), error.syscall (open, read, unlink) and error.errno. The operational distinction is this: ENOENT is usually an expected case — today's report does not exist yet — and deserves its own code branch; the rest are real failures that must be propagated. Catching them all alike is the fastest way to hide a production EACCES behind a "there was no data".

Why checking first with `exists` is a bad idea

It looks like common sense to write this:

// BAD: check before acting.
async function readIfExists(file) {
  try {
    await fs.access(file);             // does it exist?
  } catch {
    return null;                        // it does not
  }
  return fs.readFile(file, 'utf8');     // it does, read it
}

It is wrong for a deep reason. Between the access line and the readFile line there is a window of time in which the event loop does other things, and in that window another process — or your own program — can delete the file, rename it or strip its permissions. By the time readFile runs, the check is already a lie. This is the classic TOCTOU race condition (Time Of Check to Time Of Use): you check at one instant and use at another. Besides being incorrect, it doubles the work: two system calls where one was enough.

// GOOD: attempt and catch.
async function readIfExists(file) {
  try {
    return await fs.readFile(file, 'utf8');
  } catch (error) {
    if (error.code === 'ENOENT') return null;  // expected case
    throw error;                                // real failure
  }
}

The read operation already checks existence, atomically and inside the operating system. The general rule: attempt and catch, do not ask and act. That is why fs.exists has been deprecated for years (its callback did not even follow the error-first convention); fs.existsSync still exists and is legitimate at the start of a script to give a clear message, but never as a step before an operation.

Writing: `writeFile`, `appendFile` and the `flag` option

// Writes the whole file. If it exists, it TRUNCATES and replaces it.
await fs.writeFile('reports/occupancy.json', JSON.stringify(data, null, 2), 'utf8');

// Appends at the end. If it does not exist, it creates it.
await fs.appendFile('reports/audit.log', `${new Date().toISOString()} sale\n`, 'utf8');

Both accept a string or a Buffer. If you pass an unserialized object you will get the literal [object Object] in the file: serialization is always explicit, with JSON.stringify(x, null, 2) per the project convention. Behind both there is the flag option, which decides how the file is opened:

`flag`	Use	If it does not exist	If it exists
`'w'`	Write (default in `writeFile`)	Creates it	Empties it
`'a'`	Append (default in `appendFile`)	Creates it	Writes at the end
`'wx'`	Exclusive write	Creates it	Fails with `EEXIST`
`'ax'`	Exclusive append	Creates it	Fails with `EEXIST`
`'r'`	Read only	Fails with `ENOENT`	Opens it
`'r+'`	Read and write	Fails with `ENOENT`	Opens it without emptying

'wx' deserves special attention: it is the correct way of saying "create this file only if it does not exist" with no race conditions, because exclusivity is guaranteed by the operating system in a single call. It is the principle of the previous section applied to writing — attempt and catch EEXIST, do not check first — and we will use it in the next lesson so we do not overwrite the day's report.

Atomic writes: temporary file plus `rename`

writeFile is not atomic. It first truncates the file to zero bytes and then writes the new content. If the process dies between those two steps — a Ctrl+C, a power failure, the OOM killer — you are left with data/events.json empty or cut in half. You have lost the catalog.

The standard solution leans on a file system guarantee: rename within the same file system is atomic. One instant before there is the old file, one instant after the new one, with no observable moment in between.

// src/utils/atomic-write.js
// Writes a file in a way that never leaves it half-written.

const fs = require('node:fs/promises');

async function writeAtomic(file, content) {
  // The temporary file goes in the SAME directory: rename is only atomic
  // inside the same file system.
  const tempFile = `${file}.${process.pid}.tmp`;

  try {
    await fs.writeFile(tempFile, content, 'utf8');
    await fs.rename(tempFile, file);        // atomic step
  } catch (error) {
    await fs.rm(tempFile, { force: true }); // do not leave junk on the disk
    throw error;
  }
}

// Saves the modified catalog without putting the original at risk.
// Event.toJSON() returns the plain record equivalent to the seed.
async function saveCatalog(file, events) {
  await writeAtomic(file, JSON.stringify(events.map((e) => e.toJSON()), null, 2));
}

module.exports = { writeAtomic, saveCatalog };

Why it works: while the temporary file is being written, data/events.json stays intact, so any concurrent reader sees the previous version, complete and valid; the rename swaps the name in the directory in one shot, with no truncated-file window; if the process dies halfway, the worst that remains is an orphan .tmp; and the pid in the name keeps two simultaneous processes from stepping on each other's temporary file. For truly critical data there is one more step — forcing the system cache to flush to disk with sync() before renaming — but that requires the FileHandle API from the next lesson.

Memoizing the catalog so we do not re-read it

Our getCatalog() reads the file on every call. For a console script that is irrelevant; for a server handling a hundred requests per second it is a hundred disk reads of a file that does not change. The solution is memoization: read once, keep the result and return it on subsequent calls.

// src/catalog-data.js (added excerpt)

let catalogPromise = null;   // Cache: it stores the PROMISE, not the result.

async function getCatalog({ reload = false } = {}) {
  if (reload) catalogPromise = null;

  if (catalogPromise === null) {
    // We store the promise immediately, without awaiting it.
    catalogPromise = readEventsFile()
      .then((data) => data.map((record) => Event.fromJSON(record)))
      .catch((error) => {
        // A failure must not stay cached forever: clear it and propagate.
        catalogPromise = null;
        throw error;
      });
  }

  return catalogPromise;
}

The fine detail — the one that separates a correct cache from a useless one — is that we cache the promise, not the resolved value. If we stored the result (if (catalog === null) { catalog = await read(); }), ten simultaneous calls during the first await would all see catalog === null and fire ten parallel disk reads: the so-called cache stampede. By storing the promise synchronously, before any await, the second call already finds the in-flight promise and hooks onto it. A single read, even if a thousand requests arrive at once.

One side effect worth knowing: since every Event is now shared among all consumers, if somebody sells tickets on it the change is visible everywhere. That is what we want until the database of Module 7 arrives, but it also means one consumer can modify what another sees.

Common Mistakes and Tips

Using readFileSync "because it is simpler" inside a server. It is the number one cause of unexplained latency in Node: simple for you, catastrophic for your users.
Forgetting 'utf8' and being surprised by <Buffer 5b 0a ...>. If you expected text and see that, the encoding is missing. And do not compare error.message: the message is for humans, the contract is error.code.
A giant try/catch around everything. Wrap the specific operation that can fail in a predictable way, so you can tell an expected ENOENT apart from a programming error.
fs.writeFile(file, object) writes [object Object]; and writing into a non-existent directory fails with ENOENT, because writeFile does not create directories (that is mkdir with recursive, in the next lesson).
Tip: when an fs error puzzles you, print error.code, error.syscall and error.path; those three fields solve almost any mystery. And validate the data the moment you read it: a syntactically correct JSON file can carry capacity: "420" as text and break the arithmetic much later, in a place that has nothing to do with it.

Exercises

Exercise 1: seed verifier

Write src/lab/verify-seed.js that reads data/events.json with fs.promises and checks: that the file exists and is valid JSON (telling both failures apart through error.appCode); that there are 3 events and 7 sessions; that every id is unique; that no session has sold > capacity; and that the totals are the known ones (capacity 3000, sold 1811, available 1189). It must exit with process.exitCode = 1 if anything fails, print diagnostics on stderr and the summary on stdout.

Exercise 2: price increase with an atomic write

Write src/lab/raise-prices.js that reads the catalog with getCatalog(), raises the priceCents of every session of a given --venue="Sala Boveda" by --percent=10, rounding to whole cents, saves the result with writeAtomic only if --confirm is passed (without that option it just shows the changes) and appends one line per modified session to reports/price-changes.log with appendFile.

Exercise 3: measuring the cache stampede

Modify getCatalog to count how many times the file is actually read. Write two versions of the memoization — one caching the value and another caching the promise — and fire 50 simultaneous calls with Promise.all against each one. Print the number of reads of each version and explain the difference.

Solutions

Solution 1. Loading is literally the readEventsFile() from section 4, with its two separate try/catch blocks and its two error.appCode values (DATA_UNAVAILABLE and CORRUPT_DATA): reuse it instead of duplicating it. What is new is the verification:

// src/lab/verify-seed.js (central excerpt)
function verify(data) {
  const problems = [];
  const ids = new Set();
  let sessions = 0, capacity = 0, sold = 0;

  for (const event of data) {
    if (ids.has(event.id)) problems.push(`duplicate id: ${event.id}`);
    ids.add(event.id);

    for (const session of event.sessions) {
      if (ids.has(session.id)) problems.push(`duplicate id: ${session.id}`);
      ids.add(session.id);
      if (session.sold > session.capacity) {
        problems.push(`${session.id}: sold ${session.sold} > capacity ${session.capacity}`);
      }
      sessions += 1;
      capacity += session.capacity;
      sold += session.sold;
    }
  }

  if (sessions !== 7) problems.push(`expected 7 sessions, found ${sessions}`);
  if (capacity !== 3000) problems.push(`capacity ${capacity}, expected 3000`);
  if (sold !== 1811) problems.push(`sold ${sold}, expected 1811`);

  return { problems, sessions, capacity, sold, available: capacity - sold };
}

A single Set covers events and sessions because the evt- and ses- prefixes already keep them in separate namespaces. And notice that problems are accumulated instead of aborting on the first one: a verifier that only reports the first failure forces you to run it five times. main() prints the summary on stdout, dumps the problems on stderr and sets process.exitCode.

Solution 2. The structure is the one from catalog.js: read options, transform, decide. What is interesting is that dry-run mode is the default.

const events = await getCatalog();
const changes = [];

for (const event of events.filter((e) => e.venue === venue)) {
  for (const session of event.sessions) {
    const previous = session.priceCents;
    session.priceCents = Math.round(previous * (1 + percent / 100));
    changes.push({ session: session.id, previous, next: session.priceCents });
  }
}

console.table(changes);

if (!confirm) {
  console.error('Dry run. Add --confirm to write the changes.');
  return;
}

await saveCatalog(EVENTS_FILE, events);
await fs.appendFile('reports/price-changes.log', changes
  .map((c) => `${new Date().toISOString()} ${c.session} ${c.previous} -> ${c.next}\n`)
  .join(''), 'utf8');

That a script modifying data demands explicit confirmation is not a whim: it is the difference between making a mistake and being able to undo it. Watch out for the appendFile: if reports/ does not exist it fails with ENOENT, because writing does not create directories; create it by hand with mkdir reports until the next lesson.

Solution 3. The version caching the value prints 50 reads; the one caching the promise prints 1. In the first, the 50 calls come in, find the variable at null — none of them has finished yet — and all trigger their read before the first one assigns anything. In the second, the assignment of catalogPromise happens synchronously, before the first await, so call number 2 already finds an in-flight promise and simply waits for it. The general lesson: in an asynchronous cache you cache the operation in flight, not its result.

Conclusion

Escena Viva now reads from disk. src/catalog-data.js has stopped being an inlined array and become an asynchronous data layer that reads data/events.json, parses it with JSON.parse, builds Event instances with fromJSON and translates file system failures into domain vocabulary through error.appCode. The signature we designed in Module 2 survived the change without breaking anyone: all it took was an await and a boundary with .catch(). Along the way you have fixed the criteria that govern every use of fs from here on: node:fs/promises by default, the synchronous version only at startup or in one-shot scripts — and you have measured with your own eyes the 31 ms of frozen loop that ignoring it costs —, 'utf8' for text and its absence for binary, error.code instead of messages, attempt and catch instead of ask and act, the flag option to control how the file is opened, and atomic writes with a temporary file plus rename so that a power cut does not destroy the catalog. And you know that in an asynchronous cache what you store is the promise, not the value.

But we have only used two functions of fs, and the module has dozens. In the next lesson, The fs Module in Depth, we leave the single file behind: metadata with stat and the Stats object, directory walking with readdir and withFileTypes, creating and deleting whole trees with recursive mkdir and rm, the FileHandle API with its descriptors that must be closed without fail, permissions and change watching. And we will apply it to something Escena Viva already needs: a report store that organizes outputs by month, lists them and purges the old ones.

Reading and Writing Files

Contents

The three APIs of the `fs` module

`readFile`, encodings and what happens if you omit them

When the synchronous version is acceptable (and when it is a disaster)

The central task: an asynchronous data layer

Asynchrony propagates upward

File system errors and their codes

Why checking first with `exists` is a bad idea

Writing: `writeFile`, `appendFile` and the `flag` option

Atomic writes: temporary file plus `rename`

Memoizing the catalog so we do not re-read it

Common Mistakes and Tips

Exercises

Exercise 1: seed verifier

Exercise 2: price increase with an atomic write

Exercise 3: measuring the cache stampede

Solutions

Conclusion

Node.js Course: From Beginner to Advanced

Module 1: Introduction to Node.js

Module 2: Core Concepts

Module 3: File System and I/O

Module 4: HTTP and Web Servers

Module 5: NPM and Package Management

Module 6: The Express.js Framework

Module 7: Databases and ORMs

Module 8: Authentication and Authorization

Module 9: Testing and Debugging

Module 10: Advanced Topics

Module 11: Deployment and DevOps

Module 12: Real-World Projects

Reading and Writing Files

Contents

The three APIs of the fs module

readFile, encodings and what happens if you omit them

When the synchronous version is acceptable (and when it is a disaster)

The central task: an asynchronous data layer

Asynchrony propagates upward

File system errors and their codes

Why checking first with exists is a bad idea

Writing: writeFile, appendFile and the flag option

Atomic writes: temporary file plus rename

Memoizing the catalog so we do not re-read it

Common Mistakes and Tips

Exercises

Exercise 1: seed verifier

Exercise 2: price increase with an atomic write

Exercise 3: measuring the cache stampede

Solutions

Conclusion

Node.js Course: From Beginner to Advanced

Module 1: Introduction to Node.js

Module 2: Core Concepts

Module 3: File System and I/O

Module 4: HTTP and Web Servers

Module 5: NPM and Package Management

Module 6: The Express.js Framework

Module 7: Databases and ORMs

Module 8: Authentication and Authorization

Module 9: Testing and Debugging

Module 10: Advanced Topics

Module 11: Deployment and DevOps

Module 12: Real-World Projects

The three APIs of the `fs` module

`readFile`, encodings and what happens if you omit them

Why checking first with `exists` is a bad idea

Writing: `writeFile`, `appendFile` and the `flag` option

Atomic writes: temporary file plus `rename`