AlpinaShop's data platform works. Orders come in on their own, Dataflow cleanses them, Spark calculates the average basket, Data Fusion brings in the carrier's file and Workflows coordinates the lot every night with its quality check. Technically it is a success.

And yet, three things still do not work.

Nobody knows what is there. In alpinashop_analitica there are sixteen tables, six views, two materialised views and one called pedidos_evento that nobody remembers the reason for. When somebody from marketing asks "where is the conversion figure?", the answer is "ask Lucía". And if Lucía is on holiday, the answer is "I don't know".

Nobody knows what it means. The ventas_eur column appears in four tables. Does it include VAT? Does it net off returns? Does it count cancelled orders? Marta thinks it does, Lucía thinks it does not, and the report that goes to management uses one of the two definitions without saying which. Nobody has ever written it down.

Nobody knows who can see what. In 04-01, email_cliente was protected with a policy tag. But six lessons have gone by since then: Dataflow writes into pedidos_streaming, the Pub/Sub subscription dumps into pedidos_evento, the event archive goes to Cloud Storage and Data Fusion brought in the carrier's CSV with the recipient's name. Are there customer emails in any of those places without protection? Nobody has checked.

And there is a fourth missing thing, the one management has been asking for since module 3: the dashboard. The data is impeccable and nobody can see it.

In this lesson we close the module by solving all four. You will see what data governance is and why a small business needs it as much as a multinational, you will organise and document the platform with Dataplex, you will define quality rules that check themselves, you will discover and de-identify personal data with Sensitive Data Protection, and you will finally build AlpinaShop's dashboard in Looker Studio, avoiding the permissions mistake that turns a shared report into a data leak.

1. What data governance is and why a small business needs it too
2. Dataplex: lakes, zones and assets
3. The universal catalogue and data search
4. Tags and templates: documenting alpinashop_analitica
5. Data profiles: what is really in your tables
6. Quality rules and what to do when they fail
7. Automatic end-to-end lineage
8. Sensitive Data Protection: discovering where the personal data is
9. De-identification: masking and tokenisation
10. Looker Studio: connecting and modelling
11. AlpinaShop's management dashboard
12. Cost and performance: extract, live query and BI Engine
13. Permissions: the classic owner's-credentials mistake
14. Looker and LookML: when it justifies its price
15. Checklist for a healthy data platform

Data governance is the set of practices that ensure an organization's data is findable, understandable, trustworthy, secure and managed throughout its life. Five pillars:

The usual objection is that this is a matter for banks and multinationals. It is exactly the other way round, and it is worth arguing:

GDPR does not distinguish by size. A company of 40 people processing European customers' data has the same obligations as one with 40,000: knowing what personal data it holds, where it is, on what legal basis, how long it keeps it, and being able to honour an erasure request. If you do not know where a customer's email is, you cannot delete it when they ask.

The cost of not knowing is proportionally greater. In a large company there is redundancy: if somebody leaves, somebody else knows. At AlpinaShop, if Lucía leaves, knowledge of the platform leaves with her. Documentation is not bureaucracy, it is continuity.

Errors reach management sooner. With three people and one dashboard, a badly calculated figure passes through no intermediate filter. It goes straight into a decision.

And the right moment is now. Governing sixteen tables is an afternoon's work. Governing two hundred, after three years of disorderly growth, is a project of months. Governance debt accrues interest.

Dataplex is Google Cloud's unified data management layer. Its central idea: an organization's data is spread across BigQuery, Cloud Storage and other systems, and you need a logical layer above them that organises it, describes it and applies common policies to it without moving it.

Its hierarchy:

Zones come in two types with different semantics:

• Raw: data in its original format, unvalidated. Any format.
• Curated: data already cleansed and structured, with a schema. Only structured formats (Parquet, Avro, ORC) or BigQuery tables. Dataplex validates that whatever is registered here complies.

Creation:

The --discovery-enabled option is the one that pays off immediately: Dataplex automatically crawls the assets, infers schemas from the bucket's files, detects partitions from the folder structure, and creates BigQuery external tables so those files are queryable with SQL without anyone declaring them.

For AlpinaShop that means the Parquet files the Spark job leaves in gs://alpinashop-datalake/resultados/cesta/ show up as a queryable table with no extra work. It is automatic data discovery, and it is the practical reason to register the buckets even if formal governance is of no interest.

Dataplex's Universal Catalog — the heir to Data Catalog — automatically indexes the metadata of BigQuery, Cloud Storage, Pub/Sub, Spanner and Bigtable across the whole organization. And on top of that index there is a search engine.

The search syntax:

From the command line:

That query answers in seconds a question that without a catalogue requires opening table after table: where are the emails? It is the first step of section 8.

The answer to marketing's question — "where is the conversion figure?" — goes from "ask Lucía" to searching for conversion in the catalogue and finding agg_embudo_dia with its description. That change is the entire value of the catalogue.

The catalogue indexes what exists, but it does not know what it means. That is what tags are for: they add business metadata to tables and columns, structured according to a template.

First the template, which defines what fields are documented and with what permitted values:

And now each table is tagged:

That definicion_negocio field is the solution to the second problem from the start of the lesson. One sentence written once removes the argument about whether total_pedido includes VAT for good. And it sits where people search, not in a document nobody opens.

The tagging discipline AlpinaShop adopts:

The fit for reports column is the most useful one day to day: it separates what can feed a dashboard from what is plumbing. Without it, somebody will eventually build a management report on top of pedidos_staging, which is emptied every night.

Before defining quality rules you have to know what the data looks like. Dataplex's profiling analyses a table and produces per-column statistics: null values, distinct values, minimums, maximums, averages, percentiles and the most frequent values.

A typical profile reveals things nobody suspected:

Three actionable findings in a single pass: 12% of purchases with no identified customer (which may be normal or may be a registration bug), the capitalisation inconsistency in canal that breaks any GROUP BY, and a negative total_pedido, which is probably a badly modelled credit note and is distorting every average.

That last one is the perfect example of why profiling comes before rules: the rule "total never negative" is not invented, it is discovered.

With the profile in hand you define the rules. A quality scan runs a set of checks and gives a pass or fail result with the detail per rule.

The six standard quality dimensions, with their interpretation at AlpinaShop:

What to do when a rule fails. This is the part most often neglected, and without it the scan is decorative:

The integration with the orchestrator from 04-06 closes the circle:

With fail_on_dq_failure=True, a quality failure stops the nightly process before bad data spreads to the dashboard. It is the industrialised version of the quality check we wrote by hand in 04-06.

In 04-05 we saw Data Fusion's lineage, limited to its own pipelines. Dataplex Lineage does it across the board: it automatically captures lineage from BigQuery (every query that writes into a table), Dataflow, Data Fusion, Composer and Dataproc.

And that is it: from then on, every job records its own lineage.

AlpinaShop's complete graph, which Dataplex builds without anyone drawing it:

The questions this graph answers in one click:

• "The margin in the report looks odd. Where does it come from?" → agg_ventas_categoria_dia ← lineas_pedido + costes_producto_erp ← Datastream ← the ERP's MySQL. Four hops to the source.
• "We are going to change the ERP schema. What breaks?" → Everything hanging off costes_producto_erp, including the management dashboard.
• "A customer exercises their right to erasure. Where is their data?" → Trace pedidos forwards and every derived table appears.
• "Can we switch off the nightly Cloud SQL export?" → The graph shows that pedidos_staging depends on it. No.

That last question is the one that saves the most money in practice: knowing what can be switched off. Every two-year-old data platform accumulates processes that no longer feed anything, and without lineage nobody dares touch them.

GDPR warning. This section and the next deal with personal data. The tools described are technical controls, not a legal opinion. Determining the legal basis for a processing activity, the retention periods, the need for an impact assessment or whether a pseudonymisation is sufficient is a matter for a compliance professional or the data protection officer, and must be done before anything goes into production. All the data in this course is fictitious.

Sensitive Data Protection — the service formerly known as Cloud DLP — does two things: inspect, to discover where there is sensitive data, and de-identify, so that it stops being sensitive.

Let us start with inspection, which answers the third problem from the start of the lesson: are there customer emails in places where there should not be any?

An inspection job over the whole dataset:

Two deliberate decisions in that configuration:

• includeQuote: false: the findings do not include the value that was found. If you set it to true, the findings table would contain the real emails — that is, you would have created an unprotected copy of the very data you are trying to protect. It is a classic and serious mistake.
• rowsLimitPercent: 10: to discover whether there is sensitive data you do not need to scan 100%. A 10% sample finds the problem and costs a tenth as much.

And a query over the findings:

A typical result at AlpinaShop, and why it stings:

Four of the five findings are surprises. None of them is the result of bad faith: they are natural consequences of moving data around. The BigQuery subscription from 04-04 dumps the whole message. The carrier writes whatever it likes in the incident field. Customers put their phone number in a review. And an IP, which looks technical, is personal data.

This is exactly what no organization knows without inspecting. And it is the reason this section exists.

To make it continuous, Dataplex offers organization-level sensitive data profiling, which automatically scans the whole of BigQuery and maintains an up-to-date inventory of where sensitive data is and at what risk level, without launching jobs by hand.

Once the problem has been discovered, you have to act. The techniques, from least to most utility preserved:

The critical distinction is between hashing and deterministic tokenisation:

• A keyed hash produces the same result for the same value, so it allows you to count and group distinct customers, but there is no way back. It is the right choice for pure analytics.
• Deterministic tokenisation with format preservation produces a token that looks like an email and can be reversed with the key. It is the right choice when an authorised system needs to recover the original value.

De-identification configuration for AlpinaShop:

What each transformation does:

• email_cliente → hash with a key wrapped in KMS. It reuses clave-pedidos from the alpinashop-keyring keyring created in 03-06. The key never leaves KMS, so not even somebody with access to the table can reverse the hash. Lucía can still count unique customers with COUNT(DISTINCT email_hash).
• codigo_postal → masking of the last 3 characters. 08013 becomes 08XXX. The province is preserved, which is what serves geographic analysis, and the street is lost, which is what identifies.
• texto_opinion → substitution by type inside the free text. "Call me on 611223344, I am Ana" becomes "Call me on [PHONE_NUMBER], I am [PERSON_NAME]". The review is still analysable — including module 5's sentiment analysis — with no personal data inside it.

The last one is the most valuable and the one no other tool does well: de-identifying inside free text, where the personal data is not in a column but embedded in a sentence.

And the view Lucía and the dashboard are exposed to:

The operating principle: personal data exists in the base table, protected and with access restricted to gcp-seguridad@; everything else — analytics, dashboards, exploration — consumes de-identified views. Minimisation is not a one-off tidy-up, it is an architecture.

Looker Studio (formerly Data Studio) is Google's free visualisation tool. It connects to BigQuery and dozens of other sources, and lets you build interactive reports with no code.

There are three ways to connect it to BigQuery, and choosing well determines cost and performance:

The golden rule: never connect Looker Studio directly to a large detail table. Connect it to an aggregate table or a materialised view. A dashboard with six charts querying lineas_pedido runs six queries every time somebody changes a filter, and with ten users refreshing, the BigQuery bill goes through the roof — exactly the scenario in exercise 3 of 04-01.

For AlpinaShop, the dashboard's source will be a table prepared by the nightly process:

That table has a few thousand rows — two years × 3 channels × 14 countries × 6 categories — against the millions in lineas_pedido. Querying it is practically free.

Looker Studio's calculated fields are defined on the source and behave like columns:

Modelling tip: define the calculated fields in the BigQuery view, not in Looker Studio, whenever you can. A field defined in SQL is versioned in Git, is reusable by other reports and by Spark, and is not lost if somebody duplicates the report. A field defined in Looker Studio lives inside that report and nobody else sees it.

Let us design the report management has been asking for since module 3.

A three-page structure:

The key indicators, with their explicit definition — which is what avoids the "this number is not the one I had" argument:

The acquisition cost deserves an honest note: it requires a figure AlpinaShop does not yet have on the platform — the monthly advertising spend. There are two options: loading it as a manual table maintained by marketing, or integrating it from the advertising platform's API. The first is the sensible one to begin with. And it must be shown in the report stating where it comes from, because an indicator whose origin nobody knows ends up discrediting all the others.

Design good practices, which are what separate a report that gets used from one that gets opened once:

• The most important thing top left. That is where the eye goes.
• Four indicator tiles, not twelve. A dashboard with twenty figures does not get read: it gets ignored.
• Always a comparison. A number with no context does not inform. €42,180 says nothing; +12.4% against last month does.
• One chart type per question: line for trends, bars for comparing categories, table for the detail. No pie charts with twelve slices.
• Filters at the top and shared by the page, not scattered across the charts.
• Colours with meaning, and accessible: green and red make sense for good and bad, but you have to make sure they are also distinguishable for colour-blind viewers, by adding arrows or signs as well as colour.
• Last-updated date visible. A report without one breeds distrust and phone calls.
• A methodological note at the foot. Where each indicator is defined. It is the difference between a report and an argument.

The three mechanisms, in order of increasing cost:

1. Data extract. Looker Studio copies up to a data limit into its own cache and serves from there. Refresh schedulable up to once a day. BigQuery cost practically nil, excellent performance. It is the default option for a report with aggregated data that does not need to be up to the minute.

2. Live query with cache. Looker Studio caches the results for a configurable period (up to 12 hours). With the cache enabled, ten people looking at the same report generate one query, not ten. That is what should be used in a shared dashboard.

3. BI Engine. An in-memory acceleration service for BigQuery. You reserve a capacity and the queries that fit in it are answered in milliseconds and with no cost for bytes scanned.

With the bi_resumen_diario table, which takes up a few megabytes, 2 GB of BI Engine is more than enough: the whole dashboard is served from memory, with millisecond latency and no scanning charge. The reservation costs on the order of a few euros a month, and usually works out cheaper than paying for the queries it avoids.

Checking that BI Engine is being used:

If mode is FULL, the query was resolved entirely in memory. If it is DISABLED or PARTIAL, the bi_engine_statistics.acceleration_mode field explains why — usually because the query uses an unsupported function or the table does not fit.

AlpinaShop's complete recipe, summarised: small aggregate table + cache enabled + BI Engine + report connected to the de-identified view. Monthly cost of the dashboard: a few euros. Without that discipline, the same charts would cost hundreds.

Here is the most dangerous mistake in the whole lesson, and it happens constantly because the interface makes it easy.

When you create a data source in Looker Studio you have to choose which credentials the queries run with:

The disaster scenario, step by step:

1. Lucía creates the dashboard connected to BigQuery with owner's credentials. It is the convenient option, and it is what the interface suggests.
2. Lucía has, on top of access to the aggregates, read permission on the whole of pedidos.
3. Lucía shares the report as "anyone with the link" so that an external supplier can see one chart.
4. That supplier — or anyone the link is forwarded to — can explore the data with Lucía's permissions. They can add fields, change dimensions and, if the source is a table with personal data, see it.
5. Nobody in BigQuery has granted that person any permission at all. BigQuery's permissions do not protect it, because the queries are not made on their behalf.

That is a personal data breach with a notification obligation under GDPR.

The rules AlpinaShop adopts, without exception:

1. The source always points at a de-identified, authorised view, never at a table with personal data. Even if the credentials mode is the wrong one, there is nothing sensitive behind it. This is the defence that does not depend on anyone remembering.
2. Viewer's credentials on any report that leaves the data team.
3. Never "anyone with the link" for reports with business data. Always share with specific people or groups (gcp-datos@, direccion@).
4. Quarterly review of who the reports are shared with. Permissions accumulate by themselves.
5. If owner's credentials are genuinely needed — because the viewers do not and should not have BigQuery permissions — then the source has to be an aggregate view with no personal or confidential data whatsoever. No negotiation.

Verifying from BigQuery who is querying from Looker Studio:

If a single email address shows up in that list — Lucía's — when fifteen people look at the report, you are in owner's-credentials mode. It is the thirty-second check worth doing today.

Looker (without "Studio") is a different and far more expensive product. Its essential difference is LookML, a modelling language where you define once and only once what each metric and each dimension means, and every report in the company consumes those definitions.

When it justifies its price: when the cost of inconsistency exceeds the cost of the licence. That is, when there are dozens of analysts creating reports, when "sales" means different things in three departments and meetings are spent arguing over whose number is the right one, or when you need governance of the semantic model with pull request review.

For AlpinaShop, Looker Studio is the obvious answer: three people, one dashboard, a small business budget. Consistency is guaranteed by the discipline of section 10 — defining the metrics in BigQuery views versioned in Git — which is 80% of LookML's value at 0% of the price.

Organisation and discovery

• [ ] Every dataset and bucket registered in a Dataplex lake with its zones.
• [ ] Automatic discovery enabled on the Cloud Storage assets.
• [ ] Every table with a description and with governance tags.
• [ ] An explicit distinction between report-worthy tables and plumbing tables.
• [ ] Every table with a named owner.

Quality

• [ ] Profiling run over the main tables.
• [ ] Quality rules defined for completeness, uniqueness, validity, accuracy and freshness.
• [ ] Scans scheduled daily.
• [ ] The orchestrator blocks the process if a critical rule fails.
• [ ] Historical results exported to BigQuery so the trend can be seen.

Lineage

• [ ] Lineage API enabled.
• [ ] Graph reviewed after every architectural change.
• [ ] Processes with no identified consumers switched off.

Security and privacy

• [ ] Sensitive data inspection run over the whole dataset, not only where you suspect something.
• [ ] Columns with personal data tagged with an access policy.
• [ ] De-identified views as the only analytical exposure surface.
• [ ] No BI tool connected to a table with personal data.
• [ ] Hashing and tokenisation keys in KMS, never in the code.
• [ ] Review by a compliance professional or the DPO before production.

Cost

• [ ] Tables partitioned and clustered; require_partition_filter on the big ones.
• [ ] Dashboards connected to aggregates, never to detail tables.
• [ ] Report cache enabled; BI Engine if it pays off.
• [ ] Budgets and alerts by cost centre label.
• [ ] Nothing permanently switched on that does not need to be: no forgotten Dataproc clusters, no Data Fusion instances left running, no orphaned streaming pipelines.
• [ ] Periodic audit of expensive queries with INFORMATION_SCHEMA.

Operations

• [ ] Every process orchestrated, none in a cron on a VM.
• [ ] Idempotent tasks; safe reprocessing.
• [ ] Alerts that reach a specific person, not a generic mailbox.
• [ ] Code — DDL, DAGs, pipelines, flows — versioned in Git.
• [ ] Documentation of what each process does and who to notify.

Documenting only at the beginning. A catalogue with tags from a year ago is worse than none at all, because it creates unwarranted confidence. Documenting has to be part of creating the table, not a separate project.

Setting quality rules without profiling first. You end up defining invented thresholds that fail daily, everyone ignores the alerts, and the system stops being useful. Profile, look at the real data, and then define.

Quality rules that block nothing. If a quality failure neither stops the process nor notifies anybody, it is decoration.

Inspecting with includeQuote: true. It creates a findings table that contains the sensitive data you were trying to protect. That is making the problem worse.

Confusing hashing with anonymisation. An unsalted, unkeyed hash of an email is reversible by brute force — the space of plausible emails is not that large. It is still personal data for GDPR purposes. The key must be in KMS.

Connecting Looker Studio to a detail table. Every filter fires queries over millions of rows. It is the direct route to a surprise bill.

Owner's credentials plus a public link. The most serious mistake in the lesson. Anyone with the link queries with the owner's permissions.

Not putting the update date on the report. It creates distrust, phone calls, and decisions based on last week's data in the belief that it is today's.

Defining the metrics in Looker Studio instead of in BigQuery. They stay locked inside that report, they are not versioned, and when it is duplicated they diverge.

Tip: write the business definition of every metric and publish it. Half the arguments about data are arguments about definitions, not about numbers.

Tip: put a link to the catalogue in the dashboard's footer. Anyone who wants to know what a number means should be able to get there on their own.

Tip: run the sensitive data inspection today, not when there is an incident. It costs an afternoon and it almost always finds something nobody knew about.

For the alpinashop_analitica.opiniones table: (a) write the command that registers it as an asset in the lake's curated zone; (b) create the governance tag stating owner, domain, sensitivity, freshness, retention, fitness for reports and a precise business definition; (c) define a quality scan with at least five rules covering completeness, uniqueness, range validity, a set of values and a custom SQL rule; (d) explain what action you would take when each rule fails.

The texto field in opiniones is free text written by customers. Design: (a) the inspection job that discovers what types of personal data it contains, with the right options so as not to create a copy of the sensitive data; (b) the de-identification template that keeps the text analysable — for module 5's sentiment analysis — while removing the personal data; (c) the view that would be exposed to Looker Studio; (d) the GDPR warning that would accompany the proposal before it goes to production.

The management dashboard has been running for two months and there are four problems. One: it takes 40 seconds to load and the project's BigQuery cost has gone from €12 to €310 a month. Two: INFORMATION_SCHEMA shows that every query from the report is made by [email protected], even though twelve people look at it. Three: the report is shared as "anyone with the link" because it had to be shown to an external consultant. Four: management says the sales figure in the report does not match accounting's.

Diagnose each problem, state its relative severity and propose the concrete solution with the order of action.

Solution 1

(a) Registering the asset. The table is already inside the alpinashop_analitica dataset, which was registered in full as the activo-analitica asset in zona-curada. So the table does not have to be registered individually: Dataplex discovers it when it crawls the dataset. If you wanted a separate asset — for a different dataset, say — it would be:

(b) Governance tag.

sensibilidad: datos-personales and apto_para_informes: false are the two key decisions: even though the texto field looks innocuous, it is free text and therefore unpredictable.

(c) Quality scan.

(d) Action on each failure:

Solution 2

(a) Inspection.

The three decisions being assessed:

• includeQuote: false: without this, the hallazgos_dlp_opiniones table would contain the real phone numbers and names that were found. You would have created a second, unprotected copy of exactly what you are trying to protect.
• minLikelihood: POSSIBLE instead of LIKELY: in free text written by people, data appears in irregular formats ("call me on six one one…"). A low threshold produces more false positives, which is preferable to missing a real piece of data. In a structured column you would use LIKELY.
• identifyingFields: it lets you know which review contains the finding without storing the value, which makes the result actionable.

(b) De-identification template.

Why replaceWithInfoTypeConfig and not suppression. It replaces each finding with its type in square brackets:

• Original: "Very comfortable. If you have questions write to me at [email protected] or on 611223344, I am Ana Ruiz"
• Result: "Very comfortable. If you have questions write to me at [EMAIL_ADDRESS] or on [PHONE_NUMBER], I am [PERSON_NAME]"

The text is still analysable. The review keeps its structure, its approximate length and — the important part for module 5 — its sentiment: "Very comfortable" is still there. With pure suppression the syntactic coherence would be lost and sentiment analysis would degrade.

(c) Exposed view.

The nightly pipeline writes into opiniones_desidentificadas applying the template, and the original opiniones table is left with access restricted to gcp-seguridad@. Looker Studio, Lucía and module 5's future models consume the view exclusively.

(d) GDPR warning.

Customer reviews contain personal data, both in structured fields (pais combined with sku and fecha may be re-identifiable if volumes are low) and embedded in free text. This proposal applies two technical measures: automatic de-identification of the text using Sensitive Data Protection and access restriction on the original table.

Before going to production it is necessary for a compliance professional or the data protection officer to determine: the legal basis for the analytical processing of the reviews; whether the de-identification applied constitutes effective anonymisation or mere pseudonymisation — which determines whether GDPR still applies to the result; the appropriate retention period; the risk of re-identification by combining apparently non-identifying fields; and whether the intended use in module 5 (sentiment analysis) requires additional information to be given to the data subject. Automatic de-identification is not infallible: a customer who writes "I am the one who bought the blue backpack on Tuesday at the Sabadell shop" will not be detected by any predefined data type. All the data in this course is fictitious.

Solution 3

Relative severity, from highest to lowest:

Problem 3 (public link) + Problem 2 (owner's credentials) = SECURITY INCIDENT. They are not two problems: they are one single serious one. All the queries being made by Lucía means owner's credentials mode. Combined with "anyone with the link", anyone who has or receives that link queries BigQuery with Lucía's permissions, which include the pedidos table with email_cliente. If the external consultant forwarded the link, or if it ended up in an email or a chat, there is a possible personal data breach with a notification obligation. Act on this first, today, within minutes.

Problem 4 (the figure does not match accounting) = CRISIS OF CONFIDENCE. It is second in severity because it destroys the value of everything that has been built. A dashboard management does not trust stops being used, and seven lessons of work are lost.

Problem 1 (slow and expensive) = OPERATIONAL. €310 a month hurts, but it compromises neither security nor confidence. It is fixed in an afternoon.

Action in three phases:

Phase 1 — Today, within minutes: cut off the exposure.

And in parallel: notify the security lead and the DPO. The assessment of whether this constitutes a notifiable breach under GDPR is not made by the technical team. You have to document what data was accessible, for how long and who the link was shared with.

Phase 2 — This week: restore confidence in the number.

The discrepancy with accounting has three candidate causes, and you have to determine which one it is before touching anything:

Almost certainly, accounting uses definition C — net sales excluding VAT — and the report shows A. It is not a data error: it is an undocumented definition error, exactly the second problem from the start of this lesson.

The fix:

1. Agree one canonical definition with accounting and write it down.
2. Implement it in the bi_resumen_diario table, exposing both magnitudes with unambiguous names: ventas_brutas_iva_incl and ventas_netas_sin_iva.
3. Write it into the definicion_negocio tag in the Dataplex catalogue.
4. Add a visible methodological note at the foot of the dashboard.

Phase 3 — This month: cost and performance.

The diagnosis first:

The expected result: the report is connected directly to lineas_pedido or to pedidos, and each of the eight charts runs its own query every time somebody moves a filter. Twelve people × several filters a day × eight charts = thousands of queries over millions of rows.

The four fixes, in order of impact:

1. Repoint the report to bi_resumen_diario, the few-thousand-row aggregate table the nightly process produces. On its own, this measure usually divides the cost by a hundred and brings the load time down from 40 seconds to under 3.
2. Enable the report's cache with a 12-hour lifetime. Twelve people looking generate one query, not twelve.
3. Reserve 2 GB of BI Engine. The aggregate table fits entirely in memory: millisecond responses and zero bytes billed.
4. Set a daily byte quota on the project and a budget alert on the centro-coste:analitica label, so an equivalent problem is caught at €20 and not at €310.

Expected outcome: from €310/month to under €10, and from 40 seconds to under 3.

The cross-cutting lesson of the exercise: all four problems were avoidable with what we already knew. The cost, with 04-01. The permissions, with 03-04 and section 13. The definition discrepancy, with the tagging in section 4. None of them was a difficult technical failure: all four were governance failures.

Module 4 closes here, and it closes with the platform complete.

You have understood what data governance is — catalogue, lineage, quality, classification and lifecycle — and why a small business needs it as much as or more than a multinational: GDPR does not distinguish by size, knowledge has no redundancy when the team is three people, errors reach management with no intermediate filters, and governance debt accrues interest.

You have organised the platform in Dataplex: the alpinashop-lago-comercial lake with its raw zone and its curated zone, with the buckets and the dataset registered as assets and with automatic discovery that turns Spark's Parquet files into queryable tables without declaring anything. You have used the universal catalogue to answer in seconds "where is the conversion figure?" and "where are there columns called email". And you have documented with a tag template that forces you to declare owner, domain, sensitivity, freshness, retention, fitness for reports and — most valuable of all — the business definition, that one sentence written once that settles for good whether total_pedido includes VAT.

You have profiled the tables before judging them, discovering what nobody suspected: 12% of orders with no customer, five values in canal where there should have been three, and a negative total_pedido that was distorting every average. And on top of that knowledge you have defined quality rules across the six dimensions, with thresholds, with results exported and — this is what makes them useful — with an action assigned to each failure and with integration into the orchestrator so that a critical failure stops the process before bad data spreads.

You have enabled automatic lineage and you have the complete graph from Cloud SQL and the Sabadell ERP through to the dashboard, capable of answering where a number comes from, what breaks if you change a schema, where the data of a customer exercising their right to erasure is, and which process can be switched off without fear.

You have inspected with Sensitive Data Protection and found what is always found: emails in the raw Pub/Sub dump, names in the carrier's incident field, phone numbers inside customer reviews, and IPs in the browsing data. Four surprises out of five findings, none of them in bad faith. And you have de-identified with judgement: a KMS-keyed hash so you can count customers without knowing who they are, masking of the postcode while preserving the province, and substitution by type inside free text so that reviews remain analysable with no personal data inside. With the explicit warning, repeated where it belonged, that these are technical controls and that the legal decision is a matter for a compliance professional or the DPO.

And you have finally built the dashboard management was asking for: connected to a small aggregate table and not to the detail table, with a cache, with BI Engine, with calculated fields defined in SQL versioned in Git and not locked inside the report, with four indicators and not twelve, always with a comparison, with the update date visible and with a methodological note at the foot. And with the permissions lesson that on its own is worth the whole lesson: viewer's credentials, sharing with specific people, and the source always pointing at a de-identified view, because that is the only defence that does not depend on anyone remembering. You know when Looker with LookML justifies its price — when the cost of inconsistency exceeds that of the licence — and why AlpinaShop is not in that situation.

Look at what exists now that did not exist seven lessons ago. Orders come in over Pub/Sub without coupling the shop to anything. Dataflow transforms them in batch and in streaming, with event time properly understood. Dataproc computes the algorithmic work on clusters that live for four minutes. Data Fusion brings in whatever arrives dirty from outside. BigQuery stores it and answers in seconds, partitioned, clustered and without ruining anyone. Workflows orchestrates it every night with checks that block if something does not add up. Dataplex catalogues it, measures it and traces it. And Looker Studio shows it. AlpinaShop has gone from not knowing which backpack sells best in Catalonia to having a governed data platform thanks to which Lucía does not have to ask anyone for anything.

But notice what all of this has in common: it describes what has already happened. It tells you yesterday's sales, this month's funnel, this quarter's margin, which products were bought together. It is memory, and memory is enormously valuable. It is not prediction, and it is not automation.

And AlpinaShop right now has four questions that memory cannot answer. When a customer puts a set of crampons in the basket, what should the website suggest at that instant, making use of the co-occurrence matrix Spark calculated? When 60 GB of new images come into the catalogue, does somebody have to tag them by hand or can the machine say "this is a blue 40-litre backpack"? With 8,000 free-text reviews already de-identified, is anyone going to read all of them to find out what is wrong with the front panels? And with 2,400 product pages to describe, who writes them?

In module 5, Machine Learning and AI, AlpinaShop moves from describing to predicting and automating. We will start with 05-01, Vertex AI, the platform that unifies the whole model lifecycle — data, training, evaluation, deployment and monitoring — and that will connect directly to the alpinashop_analitica dataset you have just governed. Then will come AutoML for training without writing code, TensorFlow for when full control is needed, the natural language and vision APIs that answer the questions about the reviews and the images without training anything, generative AI with Gemini for the descriptions, and MLOps with Vertex AI Pipelines so that a model is not an experiment on somebody's laptop but a production system with the same discipline of idempotency, quality, lineage and cost you have applied throughout this module.

The data is already clean, governed, documented and queryable. That was not the goal: it was the requirement. Now the interesting part begins.