The third project takes you into the world of large-scale data, which we saw in Chapter 29. While previous projects built applications (a blog, an API), this one builds a data platform: a system to store, process, and analyze large amounts of information. You will combine Glue, Athena, and Redshift to create a platform capable of extracting value from data. It is a more advanced project that consolidates a highly sought-after specialty: data engineering.

The goal is to set up a system where you can gather data from different sources, prepare it, and analyze it to draw useful conclusions. Remember Chapter 29: a data platform allows a company to turn its scattered data into valuable information for decision-making.

The project combines the data services from Chapter 29, each with its role:

S3 (Chapter 5) is the data lake (subchapter 29.1): the central, cheap, and unlimited storage where you keep all raw data, in any format. It is the heart of the platform: the place where all data "lands."

AWS Glue (subchapter 29.1) catalogs the data in the data lake (creates an inventory of what is there and where) and processes/transforms it (cleans and prepares it for analysis). It is the "librarian" that organizes the lake and gets the data ready.

Amazon Athena (subchapter 29.1) lets you query the data directly in S3 with SQL, without moving it, in a serverless way. It is for flexible and ad-hoc analysis: you ask questions of your lake data and get answers.

Amazon Redshift (subchapter 29.3) is the data warehouse: for complex and recurring analyses on the most important structured data, which it loads from the data lake. This is where business reports are generated that are consulted often and must be answered very quickly.

This is how the pieces fit together, following the pattern from Chapter 29:

Data arrives in S3 (the lake); Glue catalogs and prepares it; from there, you can query it flexibly with Athena, or load the most important data into Redshift for intensive and recurring analysis. Remember that the data lake and the data warehouse complement each other (subchapter 29.3): the lake stores everything, the warehouse powers frequent analysis.

This project strengthens your mastery of data in AWS, a highly valued specialty:

💡 Expand if you want: you can enrich the project by adding Kinesis (subchapter 29.2) to ingest real-time data into the data lake, and Lake Formation (subchapter 29.4) to govern who accesses which data. This way you cover the entire Chapter 29.

Real-world example: someone interested in data engineering (a highly sought-after field) wants to consolidate what they learned in Chapter 29 with a real project. They build a platform to analyze sales data: dump sales data (from various sources, in different formats) into a data lake in S3; use Glue to catalog and prepare it; analyze it in an exploratory way with Athena ("which products sell best by region?"); and load the key data into Redshift for the monthly reports that management consults. Everything is deployed with Terraform. By building it, they truly understand how data flows through an analytical platform and the practical difference between a data lake and a data warehouse. They end up with a functional data platform and a strong profile in a highly sought-after area. The theory from Chapter 29 becomes a real skill.

• The data platform project takes you into the world of large-scale data (Chap. 29): a system to store, process, and analyze information and extract value. Build a data lake + analytics.
• Combine the pieces from Chap. 29: S3 (the data lake: all raw data, Chap. 29.1), Glue (catalogs and prepares the data, Chap. 29.1), Athena (flexible SQL queries on S3, serverless, Chap. 29.1) and Redshift (the data warehouse for complex and recurring analysis, Chap. 29.3).
• Architecture: data → S3 (lake) → Glue (catalog/prepare) → Athena (flexible queries) and/or Redshift (intensive analysis). Lake and warehouse complement each other (Chap. 29.3).
• Consolidate a highly sought-after specialty (data engineering); 💡 expand it with Kinesis (real-time, Chap. 29.2) and Lake Formation (governance, Chap. 29.4). All with Terraform.

In the last subchapter of the chapter, we will see the most ambitious project, which ties together many advanced concepts: a multi-account landing zone with Terraform and Control Tower.