The previous lesson deliberately left the product photos out of HDFS: thousands of small files that the website has to serve over HTTP, with metadata, versions and a durability that does not depend on a NameNode. Object storage was born for exactly that. It gives up directories, partial writes and rename, and in return offers a flat model of buckets and keys, an HTTP API that Amazon S3 turned into a de facto standard, "eleven nines" of durability through replication and erasure coding, and a cost per gigabyte that no network drive can match. Today it is the default store for photos, videos, backups, documents and, increasingly, the data lake itself. In this lesson we will understand its model and its API, see how durability is achieved (with a numerical Reed-Solomon example), learn to recognise when it is not the right tool, and put it to work at Kilometre Zero with MinIO in docker-compose.yml and a boto3 module with which Montblanc Dairy will upload the photo of its aged-cheese and the website will display it using presigned URLs.

1. Object, file and block
2. The object model: buckets, keys, metadata and versions
3. The S3 API as a de facto standard
4. Durability: replication and erasure coding (Reed-Solomon)
5. Consistency in object stores
6. When to use objects and when not to
7. Cost: storage, operations and egress
8. Objects at Kilometre Zero: photos, invoices and backups
9. Hands-on: MinIO, mc and services/catalog/photos.py with boto3
10. Common Mistakes and Tips
11. Exercises
12. Conclusion

There are three ways of presenting storage to a program, and it is worth telling them apart before anything else:

The "modification" row is the one that changes everything: an object is immutable. To change one byte you upload a new object under the same key. That immutability simplifies replication (two copies of an object are either identical or one of them belongs to a different version; there are no intermediate states) and it is the reason object stores scale so far and cost so little. It is also the reason for almost all of their limitations (section 6).

• Bucket: the top-level container, with a unique name (in S3, globally unique; in MinIO, unique per deployment). A bucket has a region, an access policy, a versioning configuration and lifecycle rules. Kilometre Zero will use km0-photos, km0-invoices and km0-backups.
• Key: the object's name within the bucket, a string of up to 1,024 bytes. There are no directories: photos/aged-cheese/original.jpg is a single string, and the slash means nothing to the system. Consoles and tools display "folders" by grouping on a prefix, but it is an illusion: listing "the folder photos/aged-cheese/" is a LIST operation with prefix=photos/aged-cheese/.
• Data: from 0 bytes to 5 TB (S3). Opaque: the system does not interpret the contents.
• Metadata: system metadata (Content-Type, Content-Length, Last-Modified, ETag) and user metadata (x-amz-meta-producer: montblanc-dairy, up to 2 KB in total). Metadata travels with the object and is retrieved with HEAD without downloading the data. It cannot be modified without rewriting the object (this is done by copying the object onto itself).
• Versioning: if the bucket enables it, each PUT to an existing key does not overwrite it but creates a new version with its own VersionId, and DELETE does not delete but places a delete marker on top. Earlier versions are still there and can be recovered: it is the simplest protection against accidental deletion and against a process that uploads corrupt files.
• Lifecycle: declarative rules by prefix or tag: "objects tagged kind=thumbnail that are older than 90 days move to a cold storage class", "non-current versions are deleted after 30 days", "incomplete multipart uploads are aborted after 7 days". The system applies them on its own, with no code.
• Storage classes: the same object can live in tiers with different costs and access times: standard (milliseconds), infrequent access (cheaper per GB, more expensive per read), archive (S3 Glacier: hours to retrieve, cents per TB). MinIO implements tiering to another, remote store.
• Tags: key-value pairs for classifying objects (campaign=artisan-cheese-week) and driving lifecycle rules and access policies.

Amazon launched S3 in 2006 with a simple REST API, and today practically every object store implements it: MinIO, Ceph RGW, Google Cloud Storage (interoperability mode), Backblaze B2, Cloudflare R2, Wasabi… Learn S3 and you have learnt them all. The basic operations:

Two concepts that deserve more detail:

ETag. Every object has an entity tag that changes if the content changes. For simple uploads it is the MD5 of the content ("9e107d9d372bb6826bd81d3542a419d6"), which lets you verify the integrity of an upload by comparing the local MD5 with the ETag returned. For multipart uploads it is the MD5 of the parts' MD5s followed by -N (the number of parts), so it is no use as a checksum of the whole file. It is also used in conditional reads: the website issues a GET with If-None-Match: "<etag>" and receives 304 Not Modified if the photo has not changed.

Presigned URLs. The object store should not be publicly accessible, and Kilometre Zero's website does not want to proxy every photo (it would pay for the bandwidth twice and add latency). The solution is for the catalog service, which does have credentials, to sign a download URL with an expiry (say 15 minutes) and hand it to the browser; the browser downloads straight from the store, which verifies the signature and the date. The same mechanism works for uploads: Montblanc Dairy receives a presigned PUT URL and uploads its photo directly to the bucket without going through catalog. The signature (Signature V4) is an HMAC-SHA256 of the canonical request with the secret key; the server recomputes it and compares. It is a perfect example of a capability: a portable, limited permission, with no accounts or sessions, a topic that will come up again in 06-01.

S3 advertises an annual durability of 99.999999999% ("eleven nines"): if you store 10 million objects, you can expect to lose one every 10,000 years. This is not an availability figure (that one is 99.9%–99.99%); it is the probability that the bytes still exist. It is achieved with two techniques.

Replication. N complete copies in N different failure domains (disks, servers, racks, data centres). With 3 copies you can tolerate the loss of 2, and the overhead is 200% (3 bytes stored for every useful byte). It is what HDFS does (04-02), and it is simple and fast to read (any copy will do).

Erasure coding. Instead of copying, the object is split into k data fragments and m parity fragments are computed, such that any k of the k+m fragments are enough to rebuild the object. It tolerates m losses with an overhead of only m/k. The usual algorithm is Reed-Solomon, the same family of codes used by CDs, QR codes and space communications.

The intuition, without finite-field algebra: think of k = 2 data values, d1 and d2, and m = 2 parities:

We store [d1, d2, p1, p2] on four disks. If we lose d1 and d2 (both data fragments), we have two equations and two unknowns: d2 = p2 − p1, d1 = p1 − d2. If we lose d1 and p2: d1 = p1 − d2. Any pair of survivors is enough. Reed-Solomon generalises this to any k and m, with coefficients chosen so that every subset of k fragments is solvable, working in a finite field (GF(2⁸)) so that the additions and multiplications operate on bytes without overflowing. A numerical example with integers to see that it works:

Let us compare real configurations for 1 TB of useful data:

The price of erasure coding is reconstruction: to read an object you have to gather k fragments from k nodes (more latency than reading one copy), and to repair a lost disk you have to read k fragments for every affected object (a lot of network traffic). That is why systems use it for large, cold data and keep replication (or caches, 04-05) for what is small and hot; many do both: replication for small objects and erasure coding above a certain size.

The eleven nines are calculated by combining a disk's annual failure probability (1–3%), the rebuild time after a failure (hours) and the number of simultaneous failures needed to lose data (m+1 in the same group within that window). With RS(8,4) and repairs taking hours, that coincidence is astronomically unlikely. On top of that come per-fragment checksums and scrubbing: a background process periodically rereads everything to detect silent corruption (bit rot) and repair it before failures accumulate.

For 14 years, S3 was eventually consistent for overwrites and deletes: after a PUT to an existing key, a GET could return the previous version for a few seconds; after a DELETE, a LIST could still show the key. In December 2020 Amazon announced strong read-after-write consistency for all operations: any GET, HEAD or LIST issued after an acknowledged PUT or DELETE sees the new state. In the vocabulary of 03-01, each object is linearizable, and in that of 03-02, S3 is CP (a write is not acknowledged until the quorum of replicas has it). MinIO offers the same guarantee within a deployment, and so does Ceph RGW.

Not every provider does: some S3-compatible stores, and cross-region replication setups, are eventual (the copy in the other region takes seconds or minutes). And there is an important caveat even with strong consistency: there are no transactions across objects. Uploading original.jpg and then thumb-300.jpg are two independent operations; a reader may see the first and not the second. If the application needs them to appear together, upload the thumbnails first and the "index" object that references them last, or store the reference in the database only once all the uploads have finished.

Finally, caches in front of the store (a CDN, the subject of 08-04, or a browser honouring Cache-Control) bring eventual consistency back in on their own account: replacing original.jpg does not purge the CDN. That is why common practice is not to overwrite content keys: you upload original-v3.jpg (or use a hash of the content as part of the key) and update the reference. The immutability of the object is thus extended to the key.

An object store is a good fit when the data is:

• Immutable, or replaced wholesale: photos, videos, PDFs, backups, closed event files, build artefacts, trained models.
• Large in total volume, and accessed by a known key or by prefix.
• Served over HTTP to browsers, apps or other services, often with a CDN in front.
• Tolerant of tens of milliseconds per operation.

And it is not a good fit when you need:

• File semantics: seek and partial writes, append, locks, real directories. Systems that mount S3 as a disk (s3fs, Mountpoint) work for reads, but every write rewrites the whole object.
• Atomic rename. On a file system, mv photos/tmp/x.jpg photos/aged-cheese/original.jpg is an atomic metadata operation; in S3 it is CopyObject + DeleteObject, two operations, with no atomicity and a cost proportional to the size. Moving a "directory" of a million objects means a million copies. Many processing engines (Module 5) ran into trouble with this when moving from HDFS to S3, which is why table formats that avoid renaming (Iceberg, Delta) exist.
• Low latency and lots of small operations: reading 10,000 objects of 1 KB means 10,000 HTTP requests; a database (04-04) or a cache (04-05) does it a thousand times faster.
• Frequent, large listings: LIST is paginated at 1,000 and is the slowest operation; the application should record in its database which objects exist, not discover them by listing.
• Data that changes often: every change is a new version of the complete object.

Cloud object stores charge for three things, and the third catches many teams out:

Egress is the main reason for putting a CDN in front (08-04): the CDN caches the photos at its edges, and the bucket pays only for the first delivery to each edge. It is also the reason why moving data between clouds is expensive and why the data lake (04-02) and the compute (Module 5) should be in the same region. Some providers (R2, B2) do not charge for egress, and running your own MinIO changes the model: there is no cost per GB served, but there are disks, servers and staff to operate them.

Three kinds of platform data go to the object store:

Photos are the main case. The complete flow, from Montblanc Dairy choosing a photo in its dashboard to Anna seeing it on the website:

catalog never touches the bytes of the photo: it signs, validates and stores references. This is the usual pattern, and the one that lets the store, not the service, absorb the bandwidth.

MinIO is an open-source, S3-compatible object store written in Go that deploys as a single binary. In production it runs in distributed mode (several nodes, several disks per node, automatic erasure coding); for km0/ one node with one volume is enough. We add this to docker-compose.yml:

mc (MinIO Client) is the CLI, and it works with any S3. The useful commands from your own machine:

boto3 is the official AWS SDK for Python and, thanks to MinIO's compatibility, it works just by changing endpoint_url. The module services/catalog/photos.py gathers everything catalog needs:

Notes on the less obvious parts:

• Config(signature_version="s3v4", s3={"addressing_style": "path"}): a local MinIO has no wildcard DNS, so the bucket goes in the path (/km0-photos/…) rather than in the host (km0-photos.localhost). Signature V4 is the one modern presigned URLs use.
• upload_photo checks the ETag against the local MD5: if the network corrupted anything, it is detected straight away. With put_object (a simple upload) this equality holds; with multipart it does not, as we saw in section 3. For photos over 100 MB, use s3.upload_file, which does multipart automatically with per-part retries.
• CacheControl: immutable: since we promised not to overwrite content that the CDN is going to cache, we can allow it to be cached for a year. When the dairy changes the photo, the new version will have a different VersionId and, in Kilometre Zero's practice, the website will reference original.jpg?v=<etag> so that the URL changes.
• download_url does not talk to the server: generate_presigned_url computes the signature locally with the secret key. It is instantaneous and does not use up a bucket operation. Note that ExpiresIn cannot exceed 7 days with Signature V4.
• upload_url includes ContentType and Metadata in the signed parameters: the producer must send exactly those headers, or the signature will not match. It is the way to force the upload to carry the metadata that catalog decided on.
• restore_version shows the trick for "modifying" an immutable object: copying it onto itself from the desired version, which creates a new version with the old content.
• configure_lifecycle: in MinIO, StorageClass refers to a tiering level defined with mc ilm tier add (for example, another MinIO with slow disks, or S3 Glacier); in S3 you would use STANDARD_IA or GLACIER. The abandoned-multipart rule stops you paying for orphaned parts that nobody sees in listings.

A quick two-line test:

The curl -I returns 200 OK with Content-Type: image/jpeg, ETag, x-amz-meta-producer: montblanc-dairy and Cache-Control. Wait 15 minutes and try again: 403 Forbidden with Request has expired. That is the presigned URL doing its job.

• Thinking in folders. They do not exist. LIST by prefix is slow and paginated; record in the database which keys each product has instead of listing the bucket on every page view.
• Overwriting keys that a CDN or a browser caches. The bucket will be consistent; the caches will not. Change the key (a hash or version in the name) and update the reference.
• Exposing credentials to the browser or to the producer. Never. Sign URLs from the service, with a short expiry and with the parameters you want to enforce included in the signature.
• Public buckets "to make it work". A public bucket with LIST enabled is the most common data leak in the cloud. mc anonymous set none, and presigned URLs or a CDN with signing.
• Using rename or moving large prefixes. There is no rename: it is a copy + delete per object. Design your keys so that you never have to move them.
• Trusting the ETag as the MD5 of multipart uploads. It is not. If you need a hash of the complete file, compute it and store it as metadata (x-amz-meta-sha256) or use full-object checksums (ChecksumAlgorithm).
• Leaving egress and operations out of the budget. Serving photos straight from the bucket to thousands of users costs more than storing them; the CDN (08-04) is an economic decision before it is a latency one.
• No versioning and no lifecycle. Versioning is cheap insurance against accidental deletion; lifecycle rules stop versions and abandoned multipart uploads growing without limit. Turn them on from day one.

Exercise 1. Kilometre Zero has 4,200 products with 3 photo variants each (original 1.8 MB, thumb-800 180 KB, thumb-300 35 KB) and 2,000 producers. During Artisan Cheese Week the website serves 6 million product views a day, each with one thumb-800. (a) Work out the total storage in the bucket and its monthly cost at €0.02/GB. (b) Work out the daily and monthly egress at €0.08/GB if it is served straight from the bucket. (c) What percentage of the traffic would have to be served from a CDN for the bucket's egress to drop to 5% of the previous figure, and what change to the keys makes that possible without the risk of serving stale photos?

Exercise 2. Implement generate_thumbnails(slug) in photos.py: download original.jpg into memory with get_object, generate the 800 and 300 px variants with Pillow and upload them with upload_photo, reusing the original's metadata (read it with head_object). Then describe what the website may see if a user loads the aged-cheese page just between the upload of the original and that of the thumbnails, and how you would avoid showing a broken image.

Exercise 3. With RS(k=4, m=2) over 6 disks, a 24 MB object is split into 4 data fragments of 6 MB and 2 parity fragments. (a) How many bytes does it take up in total, and what is the overhead compared with 3 replicas? (b) If two disks fail, how many MB have to be read to rebuild the object for a read, and how many to repair a single lost fragment of that object? (c) Explain why MinIO, with 6 disks, defaults to RS(3,3) (parity = half) rather than RS(4,2), and what is gained and lost.

Solution 1:

(a) Per product: 1.8 MB + 0.18 MB + 0.035 MB ≈ 2.0 MB. Total: 4,200 × 2.0 MB ≈ 8.5 GB (plus non-current versions for 30 days; let us assume an extra 20%: ~10 GB). Cost: 10 GB × €0.02 ≈ €0.20/month. Storage is irrelevant.

(b) Daily egress: 6,000,000 × 180 KB ≈ 1.08 TB/day; monthly ≈ 32 TB. At €0.08/GB: ≈ €86/day, €2,600/month. Thirteen thousand times the cost of storing it. The GET operations (6 M/day × €0.4/M ≈ €2.4/day) also exceed storage.

(c) For the bucket to serve only 5%, the CDN must answer 95% of requests from cache (hit ratio ≥ 95%), which is realistic with 4,200 small objects and Cache-Control: max-age=31536000, immutable. The condition for that one-year max-age to be safe is that the key (or the URL) changes whenever the content changes: photos/aged-cheese/thumb-800.jpg?v=<etag> or photos/aged-cheese/<etag>/thumb-800.jpg. With immutable keys you never have to invalidate the CDN; 08-04 will go into the details.

Solution 2:

(We reuse upload_photo, which already sets variant in the metadata.) Between the upload of the original and that of the thumbnails, the bucket contains original.jpg but not thumb-800.jpg: the website, which asks for a presigned URL for the thumbnail, would get a valid URL, but the GET would return 404 NoSuchKey and the browser would show a broken image. The store is consistent per object, but there is no transaction across objects (section 5). Solution: catalog does not mark the photo as "available" in km0_catalog (nor publish photo.uploaded for the website) until the generator confirms the thumbnails; in the meantime the website shows the previous photo or a placeholder image. A more robust alternative: the generator uploads the thumbnails before the reference is updated, and the reference (in the database) is the only visible "commit".

Solution 3:

(a) 4 × 6 MB + 2 × 6 MB = 36 MB for 24 useful MB: a 50% overhead. With 3 replicas it would be 72 MB (200%): erasure coding saves half the disk while tolerating the same 2 losses.

(b) To read the object you need any 4 fragments: 24 MB read from 4 disks (with the two failures, exactly the 4 survivors). To repair a single lost 6 MB fragment you have to read 4 fragments (24 MB) and recompute: the repair cost is k times the fragment size, which is the characteristic penalty of erasure coding compared with replication (where repairing 6 MB costs reading 6 MB).

(c) With RS(3,3), each object tolerates the loss of 3 disks out of 6 (half), with a 100% overhead; with RS(4,2) it tolerates 2, with a 50% overhead. MinIO defaults to the safer configuration (and with 6 disks, two simultaneous failures are not that rare during a long repair window), and lets the operator lower the parity (MINIO_STORAGE_CLASS_STANDARD=EC:2) when capacity is preferred. You gain fault tolerance and reads that need only 3 fragments; you lose the additional 50% of disk.

Object storage changes the storage contract: flat buckets and keys instead of directories, immutable objects that are replaced in full, metadata that travels with the data, and versioning and lifecycle rules that the system applies on its own. The S3 API (PUT/GET/HEAD/LIST, multipart, ETag, presigned URLs) is the standard that MinIO, Ceph and the rest implement. Eleven nines of durability are achieved by combining replication with erasure coding, where Reed-Solomon rebuilds an object from any k of its k+m fragments with an overhead of 33–50% instead of the 200% of three copies, in exchange for costlier reads and repairs. S3 and MinIO are nowadays strongly consistent per object, but there are no cross-object transactions and no atomic rename, and the caches in front of the bucket bring eventual consistency back, so good practice is not to overwrite keys. Egress, not storage, dominates the bill, and that is the economic case for the CDN we will see in 08-04. Kilometre Zero has assigned its photos to objects (km0-photos, with photos.py, direct upload by Montblanc Dairy via a presigned URL, and delivery to the website with 15-minute URLs), along with its invoices and its backups, and has set it all up with MinIO and mc in docker-compose.yml.

We now have somewhere to keep massive files (HDFS) and files that get served (objects). What is missing is the storage that drives the platform every second: small records, queried and updated with low latency, with transactions and with the consistency guarantees we chose in 03-02. That is the territory of distributed databases, where orders will migrate to Cassandra, inventory will stay on PostgreSQL, and the consistent hashing ring of 04-01 will reappear with replicas and per-operation consistency levels.