We continue in Part VII with Chapter 29: Data Platforms on AWS, where we enter the world of large-scale data: storing, processing, and analyzing huge amounts of information. We start with the central concept of modern analytics: the data lake, and how to build it on AWS with three services that work together: S3 (storage), Glue (catalog and processing), and Athena (queries). This is the foundation for extracting value from a company's data.

A modern company generates a huge amount of data from very diverse sources: sales, web logs, sensors, social networks, applications... and in different formats (tables, text, records, files). This data, if well used, is gold: it reveals patterns, helps decision-making, and powers artificial intelligence. But there is a problem:

You need a place to gather all that data and be able to analyze it together. That place is the data lake.

A data lake is a central repository where you store huge amounts of data of any type and format, in its original form, to analyze it whenever you want. The idea: first you store everything (without having to structure it in advance), and decide how to analyze it later.

Analogy: a data lake is like a large warehouse/library where you store EVERYTHING just as it arrives (documents, photos, recordings, data...), without having to classify it perfectly upon entry. The point is that you have everything in one place and, when you need to answer a question, you go and analyze it. It contrasts with a very organized warehouse where only what fits on predefined shelves is allowed in (that would be more like a data warehouse, which we’ll see in subchapter 29.3). The lake accepts everything; you’ll give it structure when you need it.

💡 Lake vs warehouse: a data lake stores raw data, of any type, and you decide the structure when you query it (“schema-on-read”). A data warehouse (Redshift, subchap. 29.3) stores data already structured and optimized for fast queries. They are complementary; each has its role.

On AWS, a data lake is typically built with three collaborating services:

S3 (Chapter 5) is the place where the data of the data lake is stored. Remember its virtues: practically unlimited storage, very cheap, durable, and supports any type of file. It’s the perfect place to dump huge amounts of data in any format. S3 is, literally, the lake: the container for all the data.

Having millions of files in S3 is not very useful if you don’t know what’s there and where. AWS Glue solves that. It does two key things:

• Catalogs the data: discovers what data is in your S3 and creates a catalog (like an index or inventory) that says what information each dataset contains, its structure, etc. This way, the data in the lake becomes locatable and understandable.
• Processes and transforms the data (ETL): allows you to clean, transform, and prepare the data (for example, convert formats, join sources), so it’s ready for analysis.

Analogy: Glue is like the librarian of the big warehouse: it goes through everything stored, creates a catalog (you know what’s there and where to find it), and, when needed, prepares and organizes materials so you can use them. Without the librarian, the warehouse would be a chaotic mess.

Amazon Athena lets you run queries (with SQL) directly on the data stored in S3, without having to move it to a database. Using Glue’s catalog, you ask questions of your data as if it were a database, and Athena answers by reading directly from S3.

The powerful part: Athena is serverless (no servers to manage, remember the philosophy from Chapter 14) and you pay only for the queries you run. It’s ideal for analyzing lake data in a flexible, on-demand way.

Analogy: Athena is like being able to ask questions directly to the library and get answers, without having to first take out all the books and bring them to another room. You ask “how many sales were there in March in Spain?” and it answers by querying the data where it is (in S3).

The trio forms a complete data lake:

S3 stores, Glue organizes and prepares, Athena queries. Together they let you gather all your company’s data and extract value from it without building complex infrastructure.

Real-world example: a retail company wants to analyze customer behavior by combining sales, web browsing, and marketing campaign data. They build a data lake: they dump all that data into S3 (each source in its own folder). Glue scans S3, catalogs what’s there, and prepares the data (unifies formats, cleans). Then, analysts use Athena to ask SQL questions like “which products are most purchased by customers who came from a certain campaign?”, querying directly on S3, without setting up any database. They discover valuable patterns that were previously invisible with the data scattered. And all serverless: they pay for S3’s cheap storage and for the queries they run, with no servers to maintain.

• Companies generate a huge amount of scattered data from various sources and formats; leveraging it requires gathering it in one place: the data lake.
• A data lake is a central repository where you store huge amounts of data of any type, in its original format, to analyze whenever you want (store everything first, decide how to analyze it later). Like a big warehouse that accepts everything.
• On AWS it’s built with three pieces: S3 (the storage: unlimited, cheap, any format — it’s the lake), Glue (catalogs what data is there and processes/prepares it — the librarian), and Athena (queries with SQL directly on S3, serverless — ask the library).
• Together: S3 stores, Glue organizes, Athena queries, letting you extract value from all your data without complex infrastructure.
• 💡 A data lake (raw data) is complemented by a data warehouse (structured data, Redshift, subchap. 29.3).

In the next subchapter we’ll see how to capture and process data that arrives in real time, continuously, with Kinesis.