Module 6 ended with a promise and a shortcoming. The shortcoming: nothing is saved on exit. Six modules of work —the catalogue, the loans, the fines, the reservations, the history— live entirely in the process memory, and when the process ends, the memory is released and everything disappears. The promise: that this module solves it.

We start with the simpler half of persistence, which is reading. Reading is simpler than writing for a practical reason: if you get reading wrong, you destroy nothing. If you get writing wrong, you can lose an entire file, and that is exactly what is going to happen to somebody in the next lesson if they do not read the table for the append parameter.

This lesson covers the complete mental model of what a file is, how it is located and how text is read from it with the two classic APIs —FileReader and Scanner—, plus the detail that causes more real incidents than anything else in the module: character encoding. By the end, BiblioTech will start up by reading its catalogue from a text file, and if that file does not exist it will start up anyway with an empty catalogue and a record in the log, applying the graceful degradation you already know how to design.

Everything opened in this lesson is closed with try-with-resources. It is the construct from 06-06 and is taken as completely known: reverse closing order, suppressed exceptions, idempotent close(). It is not explained again; it is always used.

1. Why a program needs persistence
2. What a file is: bytes on disk versus objects in memory
3. The journey of a piece of data from the application to the disk
4. Absolute and relative paths, and the working directory
5. Path separators and portability
6. The File class: the legacy API you will still run into
7. Reading with FileReader: character by character
8. Reading with Scanner over a file
9. useDelimiter and convenient parsing
10. Character encoding: UTF-8 and the accent disaster
11. FileNotFoundException versus IOException
12. Graceful degradation: BiblioTech starts with no catalogue
13. Large files and why not to read everything into memory
14. Common Mistakes and Tips
15. Exercises

The memory of a process is volatile: it exists while the process exists. When main returns, when the user closes the window or when the operating system kills the process, the JVM releases its memory and everything that was in it ceases to exist. There is nothing to recover.

This is not a defect of Java: it is the nature of RAM. A program that only works in memory is a program with no long-term memory, and that disqualifies it for almost any real use:

Look at the last row of the left-hand column applied to BiblioTech: if the application crashes with twenty active loans, those twenty loans never existed. The real library will still have twenty books off the shelf, but the system will say they are all available. The system and the world stop agreeing, which is the worst thing that can happen to a management system.

To persist means writing the state to a medium that outlives the process. In this module that medium is the file system. In a large professional system it is usually a database —you will see that with Hibernate in 11-03—, but a database is, underneath, files with a lot of engineering on top. It all starts here.

This is the idea to internalise before writing a line of code:

A file is a sequence of bytes with a name. Nothing more. It has no types, no objects, no fields. Just bytes numbered from 0 upwards.

Compare the two worlds:

When you have in memory a Book with title "Effective Java", author "Bloch" and ISBN 978-0000000001, the object occupies an area of memory with three references to three String objects. On disk you cannot store references: a memory address means nothing tomorrow, nor on another machine. You have to serialise: turn that structure into a sequence of bytes you can interpret again later. The simplest way to do it is to write text:

That line is a complete design decision. You have chosen: one material per line, fields separated by ;, a fixed field order, true/false for availability. Any program that knows that convention can read it. It is exactly what a CSV file does, and you will formalise it in 07-07.

The alternative is to let Java do the conversion for you, with the native serialisation of 07-05, which produces unreadable bytes but preserves the complete object structure. Each option has its price, and you will see it.

An important nuance about the vocabulary: people talk about text files and binary files, but that distinction does not exist on the disk. All files are binary. A "text file" is simply a file whose bytes, interpreted with a particular encoding, produce readable characters. If you open a .jpg with a text editor you will see rubbish: not because the file is different, but because you are applying the wrong interpretation. We will come back to this in section 10 and, in depth, in 07-03.

Between your String and the platter of the disk there are more layers than it seems. Knowing them explains why I/O is slow, why the buffer exists and why a close() can fail.

The points that matter along that journey:

• The expensive frontier is the system call (orange box). Every read forces a crossing from your application code into the operating system kernel and back. That context switch costs orders of magnitude more than executing a few instructions in memory. If you make one system call per character, the cost is devastating: it is exactly what happens with an unbuffered FileReader, and you will measure it in section 7.
• Decoding is a real step (the light blue box in the middle). The bytes arriving from the disk are not characters. Somebody has to decide which byte or group of bytes forms which character, and that somebody is the charset. If nobody specifies it, a default one is used, and that is where the problems of section 10 begin.
• The operating system already has its own cache (the page cache). That is why reading the same small file twice is much faster the second time: the second read never reaches the disk. This also explains why measuring I/O performance is treacherous.
• Writing does not mean "it is on the disk". When you write, the data goes into intermediate buffers. Until they are flushed, a power cut loses it. It is the central topic of 07-02.

Keep hold of the rule that governs the whole module: the fewer system calls, the better. The entire design of Java I/O —the buffers, the blocks, transferTo— exists to reduce that number.

To read a file you have to locate it, and here the first classic stumble appears: "it worked on my machine and on the server it says it cannot find the file".

An absolute path starts from the root of the file system and is self-sufficient:

A relative path does not start from the root, but from the current working directory of the process:

The key question is: relative to what? And the answer is the one that surprises everybody: relative to the directory the JVM was launched from, not to the directory where the .class is, nor to the project directory. That directory is in the user.dir system property:

Run that program from two different places and you will see the problem live:

The same program, the same file on disk, two different results. This is what happens when it works in the IDE (which launches from the project root) and fails when run from a script (which launches from somewhere else).

How this is handled professionally:

And a piece of advice that saves a lot of time: when a file "does not exist", always print the absolute path, never the one you were given. The message Cannot find 'data/catalog.txt' tells you nothing; Cannot find '/home/marta/bibliotech/target/classes/data/catalog.txt' solves the problem for you in three seconds. This is a direct application of lesson 06-04: an exception must carry the data whoever reads it needs in order to decide.

Windows uses \ as the directory separator; Linux, macOS and practically everything else use /. Java offers two constants so you do not hard-code it:

Tell the two constants apart, because they are frequently confused:

Now, the good practical news: Windows accepts / in almost every Java API. new File("data/catalog.txt") works on Windows. So, in practice:

• For short literals in the code, write / and do not overthink it: it is readable and works everywhere.
• To compose paths from pieces, do not concatenate strings: use new File(parent, child) or, better still, Path.resolve() from NIO.2, which you will see in 07-06 and is the modern, correct way.
• Never write "C:\\data\\catalog.txt" in the code. Besides not being portable, that double backslash is an endless source of typos.

java.io.File is from Java 1.0. It represents a path, not an open file: you can create a File pointing at something non-existent and nothing happens.

The most used methods:

And here is the great defect of this API, the one that prompted its replacement:

That mute false is exactly the anti-pattern that module 6 spent seven lessons eradicating: reporting a failure without saying which one. The modern API, NIO.2 (java.nio.file.Path and Files, lesson 07-06), throws specific exceptions —NoSuchFileException, AccessDeniedException, DirectoryNotEmptyException— instead of returning a boolean.

So why do we learn File? For three very practical reasons. First, because there are millions of lines of code written with it and you are going to read them. Second, because many APIs still ask for a File as a parameter. And third, because the conversion between the two worlds is trivial: file.toPath() and path.toFile(). Use NIO.2 in new code; understand File in order to read the old code.

FileReader is the most elementary class for reading text from a file. Its fundamental method is read(), and its contract has a detail that must be properly understood:

It returns an int, not a char. Why? Because it needs an extra value that is not a valid character in order to signal end of file, and that value is -1. Valid characters run from 0 to 65535; -1 collides with none of them. If the method returned char, there would be no way to tell "I have read character 0" apart from "we are done".

Let us take the loop apart, because its shape is idiomatic and you will see it everywhere:

1. reader.read() reads the next character and advances the position.
2. code = ... stores the result in the variable. In Java an assignment is an expression whose value is the assigned one; that is why it can be used inside the condition.
3. The outer parentheses are mandatory: without them, code = reader.read() != -1 would try to assign a boolean to an int and would not compile.
4. The comparison decides whether to carry on.

And now, the problem. This code is correct but slow, and it is worth understanding exactly how slow:

With a 5 MB file, on an ordinary desktop machine, the orders of magnitude are these (illustrative figures, not a formal measurement):

Two orders of magnitude of difference, and the cause is exactly the one from section 3: every unbuffered read() translates into a request to the operating system. Five million frontier crossings to read five megabytes.

The intermediate solution, without leaving FileReader yet, is to read in blocks with the overload that accepts an array:

Two critical details in that code, and both are sources of real bugs:

• read(char[]) returns the number of elements read, which may be smaller than the size of the array even though the file has not finished. Never assume it fills the buffer.
• You have to use append(block, 0, read), not append(block). If you use the unbounded version, on the last pass you will add the rubbish left in the array from the previous pass. It is a failure that only shows up at the end of the file and with apparently random content.

The definitive way of solving this —wrapping the FileReader in a BufferedReader that manages the block for you— is the topic of 07-04. Here the why is clear; there you will see the how in full detail.

You already know Scanner from module 1, reading from System.in. The same class reads from a file: all that changes is what you pass to the constructor.

The hasNextLine() / nextLine() pair is the canonical pattern and deserves a warning:

Always check with hasNextLine() before calling nextLine(). If you call nextLine() with no lines left, it throws NoSuchElementException, which is unchecked: the compiler does not warn you and it blows up at run time. It is the same Iterator contract you saw in 05-02.

Scanner does not stop at lines: it knows how to parse types, and that is where its convenience lies. Suppose this BiblioTech loans file:

It is read like this, field by field, without splitting strings by hand:

Watch out for two Scanner behaviours that cause surprises:

• nextDouble() depends on the locale. With a Spanish locale the decimal separator is the comma, so 0.25 may fail with InputMismatchException while 0,25 works. It is a failure that only appears on some machines and is baffling. The fix is to set the locale explicitly:

This same problem, applied to CSV files opened in a Spanish spreadsheet, comes back in 07-07.

• nextInt() does not consume the line break. If you mix nextInt() with nextLine(), the nextLine() will return the empty string left up to the end of the current line. It is the trap you already met with the console in module 1, and here it is identical.

An honest comparison of the two APIs seen so far:

For a twenty-line configuration file, Scanner is convenient and its slowness is irrelevant. For the catalogue file with ten thousand materials, the right answer is BufferedReader, and you will see it in 07-04.

By default, Scanner splits on whitespace. useDelimiter changes that criterion to any regular expression, which lets you read files with their own separators without splitting strings by hand.

With this catalogue file:

It is read like this:

Useful delimiters:

That last trick has its charm for very small files:

It works, but it is a workaround. Since Java 11 there is Files.readString(Path), which does the same thing explicitly and correctly, and you will see it in 07-06. Recognise the trick when you read it in somebody else's code; do not write it yourself.

And a warning that must be given now, even though it is developed in 07-07: splitting a CSV on ; or , is correct only as long as no field contains the separator. As soon as a title like "Java: the language, the machine and the ecosystem" appears inside quotes, this code silently produces misaligned fields. Lesson 07-07 implements a CSV reader that does get it right.

This section is the most important one in the lesson. Most of the I/O failures you will see in your professional career are not logic failures: they are encoding failures.

A file contains bytes. A String contains characters. Converting from one to the other requires a mapping table, and that table is the charset, or encoding.

• ASCII (1963): 128 characters, one byte each. No é, no ï, no €.
• ISO-8859-1 (Latin-1): 256 characters, one byte each. It has é, ï, ë and the rest of the Western European accented letters. It has no € and nothing outside Western Europe.
• Windows-1252: a variant of the previous one, the Windows encoding for Western Europe. Almost the same but not exactly, which generates subtle failures.
• UTF-8 (1993): covers all of Unicode with a variable length of 1 to 4 bytes. ASCII is an exact subset of it. It is the de facto standard of the web and of data interchange.

How much each character takes in UTF-8:

A direct consequence that surprises a lot of people: in UTF-8, the number of characters in a text is not the number of bytes in the file. "café" is 4 characters and 5 bytes. That is why File.length() (bytes) does not tell you how many characters there are.

You write "Naïve Set Theory" in UTF-8 and read it as ISO-8859-1. The two bytes of the ï (0xC3 0xAF) are interpreted as two independent characters:

The other way round, you write in ISO-8859-1 and read in UTF-8. The byte 0xEF of the ï does not form a valid UTF-8 sequence, and the decoder replaces it with the replacement character:

That ï and that � are the two symptoms you have to learn to recognise instantly. When you see them, do not look for the failure in your logic: it is encoding.

And there is something worse than the ugliness. Look at this case:

The data is still wrong after you read it. It will be stored wrong, compared wrong and displayed wrong. The corruption spreads.

Up to and including Java 17, when you do not specify a charset, FileReader and FileWriter use the platform default encoding, which depends on the operating system and the locale:

This means that the same program reading the same file produces different results on different machines. It is the cause of the classic "on my laptop it looks fine and on the server it comes out with question marks".

Java 18 (JEP 400) changed the default value of file.encoding to UTF-8 on all platforms, which is a big improvement. But:

• If you work with Java 17 —the reference version of this course and the most used in production— you still have the old behaviour.
• Even if your project is Java 21, code that specifies the charset is self-explanatory and does not depend on a property somebody may change with -Dfile.encoding.

Professional rule, no exceptions: always specify the charset explicitly in every text operation. It costs nothing and eliminates an entire category of failures.

Since Java 11, FileReader accepts a Charset in the constructor:

In earlier versions —and it is still the most general form, the one you will see in 07-03— the InputStreamReader bridge is used:

And Scanner accepts it too:

The constants available in java.nio.charset.StandardCharsets, guaranteed in every JVM:

Use the constants, not the strings. new FileReader(path, StandardCharsets.UTF_8) cannot fail; the version that accepts the name as a string throws UnsupportedEncodingException if you write "UTF8", "utf-8 " with a space or any other typo. It is a compile error turned into a run-time error with no advantage whatsoever.

I/O is the natural territory of the checked exceptions of module 6, and for a reason that is now obvious: the file belongs to the outside world. It may not exist, its permissions may have changed, the disk may fill up, somebody may unplug the network drive. None of that is a failure of your program, and all of it is foreseeable.

The relevant hierarchy:

What each one means in practice:

A misleading note about the name: FileNotFoundException does not only mean "it does not exist". It is also thrown if the path points to a directory or if the file exists but you do not have read permission. That is why the operating system message is so valuable and you must never discard it.

The order of the catch blocks follows the rule from 06-02, from specific to general:

Let us put it all together with the first real piece of the module. CatalogLoader reads a text file and builds materials, and —this is the important part— if the file does not exist, the application starts up anyway with an empty catalogue.

This is exactly the criterion from 06-07 for telling the recoverable from the unrecoverable: degrade when the reduced service is still correct. A library with no materials is an empty library, which is a perfectly valid state and one in which materials can be registered. There is nothing incorrect about it. It would be very different to start up without knowing the fine rate: there you cannot degrade, because we would be charging false amounts.

The file format:

And the loader:

And the application start-up now looks like this:

The three design decisions worth pointing out:

1. The path arrives as an argument, with a default value. It is the recommended strategy from section 4: whoever runs it decides where the data is.
2. load() never returns null. It returns an empty Catalog if there is no file. It is the rule 06-07 established: null is not a way of communicating anything.
3. A bad line does not abort the file. It is discarded, counted and logged. Having a ten-thousand-line catalogue fail to load because line 4,200 has a badly written year would be a dreadful policy. This is the "result object" from 06-07 applied to importing, and in 07-04 it will become a complete report.

We close with a sizing warning. It is tempting to write this:

And for a 2 KB configuration file it is perfectly reasonable. But for three years of loan history, no:

Why the factor of two: internally a Java String from Java 9 onwards stores one byte per character when all the text is Latin-1 and two bytes per character as soon as a character outside that range appears. A single emoji or one Cyrillic character in a 2 GB file doubles the memory of the whole String. And you also need the intermediate StringBuilder while you build it, so at the peak you have the content twice over.

An OutOfMemoryError is an Error, not an Exception: it is not caught, it is not recovered from and it brings the process down, with the exception of the main boundary you saw in 06-07.

The correct alternative is to process in a stream: read a line, process it, forget it, read the next one. The memory used is that of one line, not that of the file, and the consumption is constant even if the file is 50 GB:

The practical rule:

Load the whole file into memory only if you know its maximum size and it is small. Configuration, templates, files of a few KB: go ahead. Data, logs, exports, anything that grows with use: process in a stream.

That is exactly the reason BufferedReader exists, which is lesson 07-04.

• Using relative paths and not knowing what they are relative to. The classic "it works in the IDE and fails in production". Print System.getProperty("user.dir") as soon as you have doubts and pass paths as parameters.
• Reporting the error with the relative path. Cannot find 'data/catalog.txt' is useless. Always log getAbsolutePath().
• Not specifying the charset. The most expensive mistake in the module in the long run, because it does not fail: it corrupts silently. StandardCharsets.UTF_8, always, in every text operation.
• Confusing ï with ?. ï means you wrote UTF-8 and read Latin-1. ? or � means the opposite. Recognising the symptom saves you half an hour every time.
• Using strings instead of charset constants. "UTF8" compiles and blows up at run time. StandardCharsets.UTF_8 cannot be misspelled.
• Declaring char c = reader.read(). It does not compile, and for a good reason: the -1 for end of file is needed, and it does not fit in a char.
• Forgetting the parentheses in while ((c = reader.read()) != -1). Without them it does not compile. With them, it is the idiomatic loop you will always see.
• Using append(block) instead of append(block, 0, read). It adds rubbish from the previous pass on the last read. It only fails at the end of the file, which is what makes it hard to diagnose.
• Calling nextLine() without checking hasNextLine(). NoSuchElementException, unchecked, at run time.
• Mixing nextInt() and nextLine() without consuming the line break. The same trap as in module 1, identical over files.
• Trusting nextDouble() without setting the Locale. On a machine with a Spanish locale, 0.25 may fail. sc.useLocale(Locale.ROOT).
• Reading a large file character by character. Correct and a hundred times slower. Blocks or BufferedReader (07-04).
• Loading a file of unknown size into a String. OutOfMemoryError, which is not caught. Process in a stream.
• Interpreting length() == 0 as "it does not exist". An empty file gives the same result. Use exists().
• Ignoring that listFiles() can return null. NullPointerException in the for. NIO.2 solves it (07-06).
• Tip: write a single class that knows how to read each type of file. If you have new FileReader(...) scattered across twenty classes, changing the charset or the format will be a manhunt. Concentrate the I/O in the layer it belongs to, as CatalogLoader does.
• Tip: always test with a file containing é, ï and €. A test file with ASCII only will hide every encoding failure from you until it reaches production.
• Tip: test with the missing file and with the empty file too. They are the two cases most often forgotten and the two that happen most, especially on the system's first day of life.

Write a class PathDiagnostic with a method public static void diagnose(String path) that prints a complete and useful report about a path, designed so that somebody can solve a problem with it:

1. The path exactly as it was given and its absolute form.
2. The current working directory.
3. Whether it exists; and if it does not exist, whether its parent directory exists (to tell "the file is missing" from "that is not the directory").
4. If it exists: whether it is a file or a directory, whether it is readable, its size and its number of lines.
5. A final one-sentence diagnosis with the most likely cause.

Test it with a correct path, a non-existent one and a directory.

Write EncodingDetector with a method public static void compare(String path) that reads the same file three times —as UTF-8, as ISO-8859-1 and as US-ASCII— and shows the first three lines of each reading, so as to see the effect on the accented characters.

Then add public static boolean looksLikeUtf8(String path), which returns true if the file decodes without a single replacement character (\uFFFD) when read as UTF-8. Explain in a comment why that check detects "this is not UTF-8" but cannot detect "this is Latin-1 read as UTF-8 the other way round".

Prepare a test file with Naïve Set Theory, The Café Method and Zoë Blackwood, 2018.

Extend the idea of CatalogLoader to employees. Given a file like this:

Write EmployeeLoader with:

1. public LoadResult load(), returning a result object (the idea from 06-07) with the list of loaded employees and the list of errors, instead of throwing on the first failure.
2. Validation of each line: exactly 3 fields, identifier with the format EMP-NNN, non-empty name and a number of loans between 0 and Employee.MAX_CONCURRENT_LOANS.
3. Degradation if the file does not exist: an empty result with a warning, no exception.
4. Explicit charset and try-with-resources.
5. A demonstration main that prints the report with printf.

Add to the test file a line with two fields, one with the identifier X-999 and one with 9 loans, to check that all three are discarded with their reason and the good ones are loaded.

Solution 1

What this exercise teaches: the difference between detecting a failure and explaining it. The five cases in conclude correspond to five different causes that the API sums up in a single exists() returning false. Telling them apart is what turns an error message into a solution. Notice too the comment about TOCTOU: between checking exists() and opening the file anything can happen, so the prior check does not replace the catch.

Solution 2

With a test file saved in UTF-8 containing Naïve Set Theory, The Café Method and Zoë Blackwood, 2018, the output is:

The practical conclusion is the one in the comment: encoding detection is fundamentally impossible in the general case. Text editors guess it with heuristics and sometimes get it wrong. That is why no amount of cleverness in the code replaces the simple rule: always specify the charset, when reading and when writing.

Solution 3

With this test file:

The output is:

The three decisions worth pointing out:

1. The result object instead of one exception per line. Whoever imports a file wants to see all the problems at once so as to fix them and try again, not discover them one by one. It is exactly the use case 06-07 put forward for the result object.
2. split(SEPARATOR, -1). Without that -1, split discards the trailing empty fields, so EMP-001;; would give 1 field instead of 3 and the error message would be wrong. It is a classic trap that will reappear in 07-07.
3. A missing file is not a line error. It is flagged separately, because they are two different situations: "there is no data" and "there is badly written data". Confusing them produces useless reports.

You have laid the foundations of BiblioTech's persistence.

You understand what a file really is: a sequence of bytes with a name, with no types and no structure, as against an object in memory with fields and references. You know that moving from one to the other demands an explicit format decision —one material per line, fields separated by ;— and that this decision is the seed of the CSV of 07-07. And you know the complete journey of the data down to the disk, with the conclusion that governs the whole module: the system call is the expensive operation, and the entire design of I/O exists to reduce their number.

You know how to locate a file: absolute and relative paths, the role of user.dir as the origin of the relative ones, and why the same program "cannot find" the same file depending on where it is launched from. You have the five professional strategies for not depending on that, with the recommendation to pass the path as a parameter, and the rule of always logging the absolute path in error messages. You know File.separator versus File.pathSeparator and why composing paths is better than concatenating strings.

You know java.io.File —exists, isFile, canRead, length, getAbsolutePath— and, above all, the defect that condemned it: the mute booleans that say something failed without saying why, exactly the anti-pattern module 6 eradicated. You know that 07-06 replaces it with Path and Files, and that converting between the two is a toPath().

You know how to read with FileReader character by character, with the int return value and the -1 for end of file, and why that loop is two orders of magnitude slower than necessary. You know block reading with read(char[]) and its two traps: the return value that may be smaller than the array, and the append(block, 0, read) that avoids dragging rubbish along. And you know how to read with Scanner over a file, with hasNextLine/nextLine, type parsing, useDelimiter with regular expressions, and its two surprises: the Locale that changes the decimal separator and the nextInt() that does not consume the line break.

And you have the section that prevents more real incidents than any other: encoding. You know what UTF-8 is, why é takes two bytes, what exactly it means to see ï and what it means to see �, why the platform default encoding is a time bomb between machines, what Java 18 changed and why that does not excuse you from specifying it. And you know that detecting the encoding is impossible in the general case, which makes the rule absolute: StandardCharsets.UTF_8, explicit, always.

BiblioTech now reads. CatalogLoader builds the catalogue from a text file, discards badly formed lines by counting and logging them instead of aborting the whole load, translates NumberFormatException and DuplicateReferenceException into discards with context, and —most importantly— degrades gracefully: if there is no catalogue file, it starts with an empty catalogue and a WARNING, because an empty library is a correct state. It is the error policy of 06-07 applied to I/O from the very first line of code.

What remains is the missing half, and it is the frightening one. Reading is safe; writing destroys. In lesson 07-02, Writing Files, you will see FileWriter and the append parameter that decides between adding at the end and wiping the entire file, which is the most expensive mistake in the whole module; PrintWriter with the printf you have been using since module 1; the buffer and flushing, and why what you have written may not be on the disk until you close it; the atomic write pattern —write to a temporary file and rename— which stops a file being left half finished when something fails midway, a direct sibling of the compensation in finally you already know how to write; and the system line separator, with the reason why \n is not always enough. By the end of it, CatalogExporter will stop being a class that knows how to close itself but not how to write, and BiblioTech will save what it has done for the first time.