In 03-02, when a single IP address started making thousands of requests per minute against the shop,
the solution was elegant and took a minute: a DENY rule numbered 50 in the NACL of the public
subnets, with a low number so it would be evaluated before the general ALLOW. It worked perfectly.
It worked because it was one address.
If tomorrow the requests arrive from ten thousand addresses spread across sixty countries, that rule is useless. And it is not just that you would have to write ten thousand rules: it is that you cannot. A NACL takes 20 rules by default and a hard maximum of 40. A security group takes 60. The tool that solved the small problem does not scale to the big one, and that is not a defect of the tool: they are simply different problems.
Worse still, there is collateral damage that almost nobody anticipates the first time. The Auto
Scaling group asg-mercadofresco-tienda that we set up in 02-01 to solve the Friday outages will do
exactly what it was designed to do: see a lot of load and launch instances. A denial-of-service
attack against an elastic architecture does not always bring the service down; sometimes it simply
turns it into a five-figure bill.
AWS Shield is the protection service against distributed denial-of-service attacks. In this lesson Marta understands what she is protecting herself against, discovers that most of the protection is already enabled and free, and takes a reasoned —and negative— decision about the 3,000 dollars a month of Shield Advanced.
Warning. The content of this lesson is strictly defensive: it describes how to detect and mitigate attacks against your own infrastructure, never how to carry them out. Running load or stress tests against systems that are not yours is illegal, and against your own on AWS it requires following the AWS simulated testing policy. The examples are teaching material: any security configuration, and in particular any incident response plan affecting customer data under the GDPR, must be reviewed by a security professional before being applied to a real environment.
Contents
- What a distributed denial-of-service attack is
- Why the single-IP rule does not work
- Taxonomy: layers 3/4 versus layer 7
- Volumetric attacks
- Application-layer attacks
- The economic damage: when scaling plays against you
- AWS Shield Standard: what you already have
- AWS Shield Advanced: what it adds
- Cost protection against scaling
- The incident response team (SRT)
- Real cost and an honest assessment for MercadoFresco
- Resilient architecture: the first line of defence
- Reducing the exposed surface
- Caching aggressively and over-provisioning
- Metrics and detection
- Alarms towards
alertas-mercadofresco - Response plan: what Marta looks at first
- Telling an attack from a very good Friday
- What to do in the heat of the moment and what to document afterwards
- Cost, cleanup and what comes next
What a distributed denial-of-service attack is
A denial-of-service (DoS) attack aims to make a service unreachable for its legitimate users by consuming a finite resource: bandwidth, connections, CPU, memory or database connections. Distributed (DDoS) means the traffic comes from many simultaneous sources —typically compromised machines or abused third-party services—, which makes telling them apart and blocking them one by one unworkable.
The difference from other attacks matters:
| Intrusion attack | Denial-of-service attack | |
|---|---|---|
| Goal | Steal or alter data | Make the service stop responding |
| Stealth | Maximum: it wants to go unnoticed | None: it wants to be noticed |
| Defence | IAM, encryption, WAF, patches | Capacity, filtering, absorption |
| Duration | Months without being detected | Minutes or hours |
| Damage | Data breach, GDPR | Lost sales, reputation, bill |
For MercadoFresco, a two-hour attack on a Friday afternoon means losing around 1,800 orders —900 an hour at the peak— plus the damage of customers who wanted to buy and could not remembering it the following week.
Who launches them: extortion ("pay up or we carry on"), unfair competition, activism, or simply automated background noise sweeping the internet for easy targets. That last category is the most frequent and the reason even a mid-sized shop needs to think about this.
Why the single-IP rule does not work
It is worth seeing the contrast with numbers:
| The 03-02 incident | Distributed attack | |
|---|---|---|
| Sources | 1 IP address | 10,000-1,000,000 addresses |
| Solution | 1 rule in the NACL | Impossible with a NACL |
| Limit of the tool | 40 rules per NACL | Nowhere near enough |
| Where it is filtered | In the subnet, already inside your VPC | It must be filtered earlier, at the edge |
| Who decides | You, by hand, in a minute | An automatic system, in seconds |
There is also a deeper conceptual problem. Filtering in the NACL means the packet has already reached your VPC: it has consumed your inbound bandwidth and made it to the door. If the attack saturates the link, filtering inside does not help, because the link is already full.
Hence the principle that organises the whole defence against DDoS:
Malicious traffic must be filtered as far away from your infrastructure as possible, ideally in the provider's network, spread across hundreds of points of presence, before it converges on a single destination.
That is exactly what Shield does, and it explains why the protection is associated with CloudFront and Route 53: they are services that live at the edge.
Taxonomy: layers 3/4 versus layer 7
Attacks are classified by the OSI model layer they operate at, and that classification determines the defence:
| Layer 3/4 (network and transport) | Layer 7 (application) | |
|---|---|---|
| What it saturates | Bandwidth, connection table | CPU, memory, database |
| Volume | Very high: Gbps, Mpps | Low: it looks like normal traffic |
| Appearance | Malformed packets or floods | Valid HTTP requests |
| Detection | Easy: the volume gives it away | Hard: it looks like legitimate traffic |
| Defence | Absorption at the edge, signature filtering | WAF, rate-based rules, CAPTCHA |
| AWS service | Shield | WAF (04-05) |
The operational conclusion is the one that structures this module: Shield for the volumetric stuff, WAF for the clever stuff. Neither replaces the other, and the next lesson covers the second.
Volumetric attacks
The layer 3 and 4 ones. They have existed for decades and today they are almost always launched from botnets or by abusing badly configured services on the internet.
| Technique | How it saturates | Symptom in your metrics |
|---|---|---|
| UDP flood | Sends enormous amounts of UDP to random ports | Massive inbound traffic; CPU spent on filtering |
| Reflection and amplification | Spoofs your IP as the source and asks badly configured DNS/NTP/memcached servers to reply; the reply is hundreds of times bigger than the request | Enormous volume from legitimate servers' IPs |
| SYN flood | Opens TCP connections halfway and never completes them, exhausting the connection table | Many connections in SYN_RECV; the server accepts no new ones |
| ICMP flood | Massive ping | High inbound traffic, not very effective today |
| Fragmentation | Fragmented packets that consume resources on reassembly | High CPU in the operating system |
Reflection amplification is what produces the largest attacks on record: an attacker with little bandwidth can generate hundreds of times that amount against their target. The mitigation is a network one and happens before it reaches you: it is not something you solve on your server.
The good news for MercadoFresco is that AWS absorbs nearly all of this by default. The aggregate capacity of the AWS network and of the CloudFront points of presence is of an order of magnitude no volumetric attack aimed at a mid-sized shop is going to exhaust. Against this family, MercadoFresco is already protected without having done anything.
Application-layer attacks
This is where a company the size of MercadoFresco has a real problem, because these attacks do not need volume.
| Technique | How it works | Why it hurts |
|---|---|---|
| HTTP GET flood | Thousands of requests per second to ordinary pages | Every request consumes CPU and a connection |
| HTTP POST flood | Requests that force writes or processing | Even more expensive than GETs |
| Slowloris | Opens many connections and sends the headers very slowly, without closing them | Exhausts the server's connections with very little traffic |
| Attack on search | Complex, uncacheable queries against the search engine | One request can cost seconds of database time |
| Credential stuffing | Tries leaked user/password pairs against /login |
As well as load, it is looking for access |
| Basket abuse | Adds and removes products, reserving stock | Consumes database and locks up inventory |
The specific case that worries Marta most: product search. A request to
/buscar?q=tomate&filtros=ecologico,granel&orden=precio is not served from cache, runs a complex
query against mercadofresco-pedidos and takes about 200 ms. Fifty requests per second from fifty
different addresses —a ridiculous volume, indistinguishable from normal traffic on a bandwidth
chart— can saturate the database without a single network alarm batting an eyelid.
That is why Shield alone is not enough and why lesson 04-05 exists.
The economic damage: when scaling plays against you
This section deserves a calculation, because it is the argument that makes a company's management understand the problem.
Suppose a layer 7 attack sustained for 6 hours against MercadoFresco, with 5,000 requests per second reaching the ALB:
| Effect | Detail | Cost |
|---|---|---|
| The ASG scales to the maximum | From 2 to 20 t3.medium instances for 6 h |
18 × 0.0456 × 6 ≈ 4.92 USD |
| Outbound transfer | If the attack asks for uncached content, 500 GB | 500 × 0.085 ≈ 42.50 USD |
| ALB capacity units | Thousands of new connections per second | Tens of USD |
| CloudFront requests | 108 million requests in 6 h | 108 × 0.0075 ≈ 0.81 USD |
| RDS reads | May require scaling the instance up | Variable |
| Lost sales | 6 h of a Friday × 900 orders/h × margin | By far the most expensive |
The infrastructure cost of an attack like that on MercadoFresco is tens or a few hundred dollars: annoying, not catastrophic. The real damage is the lost sales and the trust. This calculation matters because it is what prevents the emotional decision of buying Shield Advanced at 3,000 USD a month to protect against a 200 USD risk.
In bigger architectures the calculation changes radically, and that is why the cost protection we will see two sections from now exists.
AWS Shield Standard: what you already have
Shield Standard is on, it is free and there is nothing to do to have it. It is applied automatically to every AWS customer.
What it includes:
- Automatic detection and mitigation of the most common volumetric and connection-state attacks at layers 3 and 4, in real time and continuously.
- Inline mitigation, without diverting the traffic or introducing perceptible latency.
- Protection built into CloudFront, Route 53, Global Accelerator and Elastic Load Balancing, which are precisely the services where MercadoFresco's edge lives.
- Network defences applied in the AWS infrastructure: filtering of malformed packets, per-source limits, protection against SYN floods.
What matters is where it acts:
flowchart LR
A["Internet traffic<br/>legitimate + attack"] --> B["AWS network<br/>Shield Standard"]
B -->|"malformed packets,<br/>L3/L4 floods"| X["Dropped at the edge"]
B -->|"clean traffic"| C["CloudFront<br/>E2QWERTY123ABC"]
C --> D["ALB<br/>alb-mercadofresco-tienda"]
D --> E["ASG<br/>asg-mercadofresco-tienda"]
E --> F["RDS<br/>mercadofresco-pedidos"]
The filtering happens before CloudFront, that is, before the traffic reaches anything of yours and before it generates cost. What Shield Standard does not do is tell a legitimate HTTP request from a malicious one: that is layer 7 and it is WAF's job.
A practical check: since MercadoFresco already serves all its traffic through CloudFront (03-04) and resolves DNS with Route 53 (03-05), it is already using Shield Standard in its most effective form without having consciously decided to. That is one of the non-obvious reasons for putting a CDN in front, even when the transfer saving was not the main argument.
AWS Shield Advanced: what it adds
| Feature | Standard | Advanced |
|---|---|---|
| Automatic L3/L4 mitigation | Yes | Yes, more aggressive |
| Automatic layer 7 protections | No | Yes, it writes WAF rules on its own |
| Per-resource specific detection | No | Yes, it learns your normal traffic pattern |
| Attack metrics and diagnostics | No | Yes, AWS/DDoSProtection |
| Cost protection | No | Yes, credits for the scaling during the attack |
| Response team (SRT) | No | Yes, 24/7 |
| WAF included at no extra cost | No | Yes, on the protected resources |
| Cross-account Firewall Manager | Paid separately | Included |
| Global event dashboard | No | Yes |
| Cost | 0 USD | 3,000 USD/month + transfer |
The three features that genuinely justify the price:
1. Per-resource specific detection. Shield Advanced establishes a baseline of the normal traffic of your application —not of the AWS average— and detects deviations. It can identify a 200 Mbps attack against a resource that normally receives 20 Mbps, a volume that in absolute terms would not catch anybody's attention.
2. Automatic application-layer protections. With this enabled, Shield Advanced writes and applies WAF rules on its own during an attack, based on the detected pattern, and withdraws them when it passes. It is the only way to respond to a layer 7 attack in seconds in the middle of the night with nobody awake.
3. Cost protection, which deserves its own section.
Cost protection against scaling
It is the least known feature and the one that has justified the purchase most often.
During an attack, your elastic infrastructure reacts: the ASG launches instances, CloudFront serves requests, the ALB processes connections, outbound transfer shoots up. All of that is billed.
Shield Advanced offers service credits for the charges attributable to a verified attack on the protected resources:
| Service | What is covered |
|---|---|
| CloudFront | The attack's outbound transfer and requests |
| Route 53 | The attack's queries |
| ELB | Capacity units consumed |
| EC2 | Instances launched by the scaling during the attack |
| Global Accelerator | Transfer |
How it works in practice: you open a support case during or after the attack, AWS verifies there was a DDoS event on that resource in that window, and applies a credit to the bill. It is not automatic: you have to ask for it.
For a company whose bill during an attack can go from 5,000 to 80,000 dollars in one night, this feature alone pays for the service. For MercadoFresco, whose most expensive imaginable attack costs around 200 dollars, it does not.
The incident response team (SRT)
The Shield Response Team is a team of AWS engineers specialising in DDoS, available 24/7 to Shield Advanced customers with a Business or Enterprise support plan.
What it does:
- During an attack: it investigates with you, writes bespoke mitigations, applies WAF rules in your Web ACLs if you have given it permission.
- Proactively: if you authorise it, it can step in without waiting for you to call, as soon as its alarms fire. This is what is known as proactive engagement and it has to be enabled with the contacts configured.
- Beforehand: it reviews your architecture and recommends changes.
The part to weigh up is that giving it access to your Web ACLs means authorising a third party to modify your filtering rules in production without your prior involvement. For many organisations that is exactly what they want at four in the morning; for others it is a governance issue that has to be documented.
Real cost and an honest assessment for MercadoFresco
| Item | Price |
|---|---|
| Shield Standard | 0 USD |
| Shield Advanced | 3,000 USD a month, with a 12-month commitment |
| Data transfer from protected resources | An additional per-GB rate (CloudFront, ELB, EC2) |
| Requirement for the proactive SRT | Business support (minimum 100 USD/month) or Enterprise |
| WAF on protected resources | Included |
That is 36,000 USD a year, plus transfer, plus support. The annual commitment means you cannot buy it "just while the scare lasts".
Let us do the honest assessment for MercadoFresco:
| Factor | MercadoFresco's situation | Does it push towards Advanced? |
|---|---|---|
| Annual turnover | Spanish SME, a single country | No |
| Cost of a 6 h attack | ~200 USD of infrastructure + lost sales | No |
| Exposed surface | Everything behind CloudFront, private origins | No |
| Target profile | Grocery shop, no political profile | No |
| Contractual or regulatory requirement | None | No |
| 24/7 on-call team | Does not exist | Yes, a little |
| Regulated data | Personal data, but DDoS does not expose it | Neutral |
Conclusion: MercadoFresco should not buy Shield Advanced today. The cost is roughly equivalent to its entire AWS bill multiplied several times over, to protect against a risk whose direct impact is two orders of magnitude smaller.
What it should do, and this is the rest of the lesson:
- Make the most of Shield Standard, which it already has, by serving everything through CloudFront.
- Design the architecture to absorb, not to resist.
- Set up WAF with rate-based rules (04-05), which costs a few dollars a month and covers 90 % of what worries it.
- Have alarms and a written plan.
And when it would make sense to reconsider:
- If MercadoFresco grew to turn over millions and an hour of downtime cost tens of thousands.
- If it received an extortion attack with a threat to repeat.
- If a corporate customer or a regulation required it contractually.
- If it operated in a sector with a high target profile (banking, gambling, media, public sector).
It is a business decision with numbers, not a technical decision. And knowing how to argue the "no" with data is as valuable as knowing how to configure the "yes".
Resilient architecture: the first line of defence
Ahead of any protection service, the best defence against DDoS is an architecture that absorbs. These are the four principles, and MercadoFresco already meets almost all of them:
flowchart TD
subgraph L1["Layer 1: the edge absorbs"]
A["Route 53<br/>Shield Standard"] --> B["CloudFront E2QWERTY123ABC<br/>~700 points of presence<br/>Shield Standard + cache"]
end
subgraph L2["Layer 2: intelligent filtering"]
B --> C["AWS WAF<br/>managed + rate-based rules<br/>Covered in 04-05"]
end
subgraph L3["Layer 3: minimum surface"]
C --> D["ALB alb-mercadofresco-tienda<br/>sg-mercadofresco-alb: 443 only"]
D --> E["ASG asg-mercadofresco-tienda<br/>private subnets app-a/-b<br/>sg accepts the ALB only"]
end
subgraph L4["Layer 4: isolated data"]
E --> F["RDS mercadofresco-pedidos<br/>subnets datos-a/-b<br/>no internet egress"]
B -.->|"OAC oac-mercadofresco-catalogo"| G["S3 mercadofresco-catalogo-fotos<br/>public access block"]
end
What makes this architecture strong is not any security product, but four decisions taken back in modules 2 and 3:
| Principle | How MercadoFresco meets it | Lesson |
|---|---|---|
| Serve from the edge | All traffic comes in through CloudFront | 03-04 |
| Reduce the surface | S3 closed with OAC, instances in private subnets, RDS with no egress | 03-01, 03-04 |
| Over-provision and autoscale | ASG from 2 to 20 instances across two AZs | 02-01 |
| Cache aggressively | Photos and catalogue pages in the CloudFront cache | 03-04 |
Reducing the exposed surface
The rule is simple: what is not exposed cannot be attacked. A review of MercadoFresco's surface:
| Resource | Exposed? | Comment |
|---|---|---|
CloudFront E2QWERTY123ABC |
Yes, on purpose | It is the entry point; it is designed to absorb |
alb-mercadofresco-tienda |
Yes, with public DNS | Improvable: see below |
| ASG instances | No | Private subnets, no public IP |
mercadofresco-pedidos |
No | Data subnets, no route to the internet |
mercadofresco-catalogo-fotos |
Not directly | Closed with OAC since 03-04 |
| API endpoint | Yes, via CloudFront | Protected by WAF in 04-05 |
The only outstanding improvement is the second row. Even though the public DNS points at CloudFront, the ALB still has a resolvable public DNS name, and an attacker who discovers it can bypass CloudFront and hit it directly, avoiding the cache and the WAF. The standard mitigation has two parts:
- Have CloudFront add a secret header to every request towards the origin.
- Have the ALB reject any request that does not carry it, through a listener rule.
# 1. CloudFront sends a custom header to the origin 'origen-tienda-alb'
# (configured in the distribution's origin definition)
# X-Origen-Verificado: <random value stored in Secrets Manager>
# 2. The ALB listener only lets through the ones that carry it
aws elbv2 create-rule \
--listener-arn arn:aws:elasticloadbalancing:eu-west-1:111122223333:listener/app/alb-mercadofresco-tienda/50dc6c495c0c9188/abc \
--priority 1 \
--conditions '[{"Field":"http-header",
"HttpHeaderConfig":{"HttpHeaderName":"X-Origen-Verificado",
"Values":["fictitious-rotatable-value"]}}]' \
--actions '[{"Type":"forward","TargetGroupArn":"arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/tg-mercadofresco-tienda/abc123"}]' \
--profile mercadofresco-dev
# 3. The listener's default rule returns 403
aws elbv2 modify-listener \
--listener-arn arn:aws:elasticloadbalancing:eu-west-1:111122223333:listener/app/alb-mercadofresco-tienda/50dc6c495c0c9188/abc \
--default-actions '[{"Type":"fixed-response",
"FixedResponseConfig":{"StatusCode":"403",
"ContentType":"text/plain",
"MessageBody":"Direct access not allowed"}}]' \
--profile mercadofresco-devThat header value is exactly the sort of rotatable secret that goes to Secrets Manager, as we
saw in 04-03. And to complete the lockdown you can restrict the inbound rules of
sg-mercadofresco-alb to CloudFront's published IP ranges, using the managed prefix list
com.amazonaws.global.cloudfront.origin-facing:
aws ec2 describe-managed-prefix-lists \
--filters "Name=prefix-list-name,Values=com.amazonaws.global.cloudfront.origin-facing" \
--query 'PrefixLists[0].PrefixListId' --output text --profile mercadofresco-devWith those two measures, the ALB stops being a reachable target and all the traffic is forced to go through the edge, which is where the protection lives.
Caching aggressively and over-provisioning
Caching. A request served from the CloudFront cache does not reach your infrastructure, does not consume CPU, does not touch the database and does not count towards scaling. With the hit rate we measured in 03-04, the vast majority of the catalogue traffic stays at the edge. Specific recommendations against DDoS:
- Cache error responses too (404, 403) for a few seconds. Otherwise an attack asking for non-existent routes is an attack that reaches the origin in full.
- Define a sensible minimum TTL so an attacker cannot invalidate the cache by adding random query parameters: the cache key must include only the parameters that really change the response, as we configured with the cache policies.
- Serve a static maintenance page from S3 as the Route 53 failover target, already prepared in 03-05: if everything fails, customers see something.
Over-provisioning. Absorbing is cheaper than falling over:
- Make the ASG's minimum capacity more than strictly necessary. MercadoFresco has 2 instances for 900 orders/hour when one serves 600.
- Make the scaling react fast: a short warm-up period and low thresholds.
- Give the maximum a deliberate ceiling. This point is counter-intuitive but important: an ASG with a maximum of 100 during an attack is a bill. A maximum of 20 means the service degrades, but in a bounded and predictable way. Choose the number deliberately, do not leave it at the default.
- Use instance types with guaranteed network performance if the traffic is high: the families with network credits can be throttled exactly when they are needed most.
Metrics and detection
With Shield Advanced you get the AWS/DDoSProtection namespace in CloudWatch:
| Metric | What it measures |
|---|---|
DDoSDetected |
1 if there is an attack in progress against the resource, 0 if not |
DDoSAttackBitsPerSecond |
Volume of the attack in bits per second (L3/L4) |
DDoSAttackPacketsPerSecond |
Packets per second |
DDoSAttackRequestsPerSecond |
Requests per second (layer 7) |
Without Shield Advanced —MercadoFresco's case— those metrics do not exist and you have to detect through indirect signals. These are the ones Marta watches:
| Metric | Service | Alarm signal |
|---|---|---|
RequestCount |
ALB | A sudden 5× rise over the hourly average |
TargetResponseTime |
ALB | It rises while RequestCount rises |
HTTPCode_ELB_5XX_Count |
ALB | Errors from the balancer itself: saturation |
HTTPCode_Target_5XX_Count |
ALB | The targets cannot keep up |
ActiveConnectionCount |
ALB | Many connections with few requests: slowloris |
NewConnectionCount |
ALB | An anomalous rate of new connections |
CPUUtilization |
ASG | At 100 % on every instance at once |
GroupInServiceInstances |
ASG | Scaled to the maximum outside the usual hours |
DatabaseConnections |
RDS | Close to the parameter group's limit |
Requests and 4xxErrorRate |
CloudFront | A spike with a low cache hit rate |
CacheHitRate |
CloudFront | A sharp drop: somebody is dodging the cache |
That last row is especially revealing. A legitimate traffic spike maintains or improves the hit
rate, because a lot of people ask for the same popular pages. An attack designed to do damage asks
for random routes or ones with unique parameters, and sinks the hit rate. A chart of
CacheHitRate falling while Requests rises is one of the cleanest signals there is.
Alarms towards alertas-mercadofresco
These are the alarms MercadoFresco sets up today, without Shield Advanced. CloudWatch is covered in depth in 05-01; here the mechanics are enough:
# 1. Anomalous request spike on the ALB
aws cloudwatch put-metric-alarm \
--alarm-name mercadofresco-alb-peticiones-anomalas \
--alarm-description "Possible attack: requests far above normal" \
--namespace AWS/ApplicationELB \
--metric-name RequestCount \
--dimensions Name=LoadBalancer,Value=app/alb-mercadofresco-tienda/50dc6c495c0c9188 \
--statistic Sum --period 60 --evaluation-periods 3 \
--threshold 60000 --comparison-operator GreaterThanThreshold \
--alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
--treat-missing-data notBreaching \
--profile mercadofresco-dev
# 2. Cache hit rate drop: somebody is dodging CloudFront
# (CloudFront metric: always in us-east-1)
aws cloudwatch put-metric-alarm \
--alarm-name mercadofresco-cdn-aciertos-bajos \
--alarm-description "The hit rate is falling: possible attack with random routes" \
--namespace AWS/CloudFront --metric-name CacheHitRate \
--dimensions Name=DistributionId,Value=E2QWERTY123ABC Name=Region,Value=Global \
--statistic Average --period 300 --evaluation-periods 2 \
--threshold 50 --comparison-operator LessThanThreshold \
--alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
--region us-east-1 --profile mercadofresco-dev
# 3. Scaled to the maximum: the sign that something is very wrong (or very right)
aws cloudwatch put-metric-alarm \
--alarm-name mercadofresco-asg-al-maximo \
--alarm-description "The ASG has reached 18 instances or more" \
--namespace AWS/AutoScaling --metric-name GroupInServiceInstances \
--dimensions Name=AutoScalingGroupName,Value=asg-mercadofresco-tienda \
--statistic Maximum --period 300 --evaluation-periods 1 \
--threshold 18 --comparison-operator GreaterThanOrEqualToThreshold \
--alarm-actions arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco \
--profile mercadofresco-devTwo details from the field:
- Alarm 2 is created in
us-east-1, because CloudFront always publishes its metrics there, exactly as happened with the ACM certificates in 03-04. --treat-missing-data notBreachingavoids false alarms in the small hours when there is no traffic.
And one alarm that is not a security one but saves the bill: the budget with notifications we created in 01-02. If daily spend shoots up, somebody finds out even if nobody is watching the panels.
Response plan: what Marta looks at first
A response plan exists so that you do not have to think at three in the morning. This is MercadoFresco's, in order:
flowchart TD
A["Alarm on alertas-mercadofresco"] --> B["1. Confirm: does the site<br/>work from outside?"]
B --> C["2. Dashboard: RequestCount,<br/>CacheHitRate, 5XX, CPU"]
C --> D{"3. Attack or<br/>legitimate spike?"}
D -->|"Legitimate spike"| E["Raise the ASG maximum,<br/>tell the business, enjoy"]
D -->|"Attack"| F["4. Which layer?<br/>High volume = L3/L4<br/>Normal volume = L7"]
F -->|"L3/L4"| G["Shield Standard is already acting.<br/>Check CloudFront and wait"]
F -->|"L7"| H["5. Identify the pattern:<br/>routes, countries, user agents"]
H --> I["6. Apply WAF rules<br/>in Block mode (04-05)"]
I --> J["7. Monitor the effect"]
J --> K["8. Withdraw the temporary<br/>rules when it is over"]
K --> L["9. Written post-mortem"]
Step 1: confirm from outside. Before anything else, check the problem is real and not a monitoring failure:
curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://mercadofresco.example/salud
dig +short mercadofresco.exampleStep 2: the four charts. The ALB's RequestCount, CloudFront's CacheHitRate,
HTTPCode_Target_5XX_Count and the ASG's CPUUtilization. With those four you almost always know
what is going on.
Step 5: identify the pattern. The access logs we store in mercadofresco-registros-web
since 03-03 are the source:
aws s3 sync s3://mercadofresco-registros-web/alb/2026/08/02/ /tmp/registros/ \
--profile mercadofresco-dev
# The 20 IPs with the most requests
zcat /tmp/registros/*.gz | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20
# The most requested routes
zcat /tmp/registros/*.gz | awk '{print $13}' | sort | uniq -c | sort -rn | head -20
# The most frequent user agents
zcat /tmp/registros/*.gz | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -10That analysis produces the three answers you need in order to write a WAF rule: where it comes from, what it asks for and what identifies it.
Telling an attack from a very good Friday
This is the hard part, and the one that separates a professional response from a self-inflicted blackout. MercadoFresco has legitimate, predictable traffic peaks on Friday afternoons. Blocking real traffic on a Friday at 19:00 is worse than the attack.
| Signal | Legitimate spike | Attack |
|---|---|---|
| Timing | Friday afternoon, campaigns, public holidays | Any time, typically the small hours |
| Curve | Rises over minutes or hours | Rises in seconds, vertically |
| Cache hit rate | It holds or improves | It collapses |
| Geographic distribution | Spain, a little Portugal | Countries with no customers, very spread out |
| Routes requested | Home page, categories, popular product pages | Odd, random routes, always the same face |
| Conversion to order | Normal, ~2 % | Close to zero |
| User agents | Real browsers, varied | A few repeated values, or absent |
| Referrer | Search engines, social, direct | Empty or spoofed |
| Requests/session ratio | 10-30 pages | Thousands from the same IP |
| Effect on the database | Rises proportionally | Shoots up with no orders to justify it |
The most reliable signal of all is conversion. If 5,000 requests per second arrive and the number
of orders per hour is still 900, they are not customers. A legitimate spike moves both metrics at
once; an attack only moves one. Marta has that business metric published in the MercadoFresco/Tienda
namespace thanks to the cloudwatch:PutMetricData permission we granted in 04-01, and it is the one
she looks at before deciding to block anything.
And the operational golden rule:
When in doubt, start by counting, not blocking. WAF's
Countmode lets you see who a rule would affect before applying it. It is the central topic of 04-05 and the most expensive mistake in this discipline is skipping it.
What to do in the heat of the moment and what to document afterwards
Measures in the heat of the moment, from least to most intrusive:
| Measure | Impact on customers | When |
|---|---|---|
| Raise the ASG maximum | None; cost | Whenever the load is absorbable |
| Increase the cache TTLs | Slightly older content | Immediate, very effective |
WAF rule in Count on the pattern |
None | Always first |
Rate-based rule in Block |
Blocks whoever exceeds the threshold | If the pattern is clear |
| Geolocation blocking | Blocks entire countries | If you have no customers there |
CAPTCHA or Challenge on expensive routes |
Friction for humans | An alternative to blocking |
| Static maintenance page | Degraded service | Last resort |
Never: switch off CloudFront or point the DNS at the origin. That removes the only layer protecting you and turns an incident into a total outage.
Post-mortem. When it is over, you write it up. No culprits, with data:
- Timeline with exact times: first sign, first alarm, first action, mitigation, recovery.
- Characterisation: peak volume, number of sources, countries, target routes, layer.
- Impact: minutes of degradation, estimated lost orders, extra infrastructure cost.
- What worked and what did not: did the right alarm fire? in time? was anybody there?
- Concrete actions with an owner and a date: WAF rules that become permanent, new alarms, adjusted thresholds.
- Credit request if you had Shield Advanced.
Point 5 is what turns an incident into an improvement. A post-mortem with no dated actions is a document nobody will ever read again.
Cost, cleanup and what comes next
| Item | Cost for MercadoFresco |
|---|---|
| Shield Standard | 0 USD — already on |
| Shield Advanced | 3,000 USD/month — not purchased, reasoned decision |
| CloudWatch alarms | 3 × 0.10 USD = 0.30 USD/month |
| Storage of the ALB logs in S3 | Already accounted for in 03-03 |
| ALB listener rule | No extra cost |
| Total for this lesson | 0.30 USD/month |
If you had enabled Shield Advanced to try it out, bear firmly in mind that the commitment is 12
months: it is not cancelled just like that. It is managed from the Shield console or with
aws shield disassociate-drt-role and the subscription cancellation process, which requires opening
a support case.
To undo what was created in this lesson:
aws cloudwatch delete-alarms \
--alarm-names mercadofresco-alb-peticiones-anomalas mercadofresco-asg-al-maximo \
--profile mercadofresco-dev
aws cloudwatch delete-alarms --alarm-names mercadofresco-cdn-aciertos-bajos \
--region us-east-1 --profile mercadofresco-devThe listener rule with the verified header is not deleted: it is a permanent improvement to the security posture and it should stay.
Common Mistakes and Tips
Believing Shield Standard has to be enabled. It is already on, in every account, free. What is your decision is making the most of it, by serving the traffic through CloudFront and Route 53 instead of exposing the origin directly.
Thinking Shield protects against everything. Shield is layers 3 and 4. Layer 7 attacks —the ones that really threaten a shop— are filtered by WAF. Buying Shield Advanced and not configuring WAF is spending 3,000 dollars and staying exposed to what is most likely to happen to you.
Leaving the ALB directly reachable. If the balancer's DNS name is resolvable and it accepts traffic from any source, an attacker can bypass CloudFront, the cache and the WAF in one hop. The verified header plus the CloudFront prefix list close that door.
Leaving the ASG maximum at a very high number "just in case". During an attack you will scale without limit and the result will be a bill, not an available service. Set a deliberate ceiling.
Blocking by IP during a distributed attack. It is exhausting, useless, and you end up blocking real customers sharing a corporate NAT. Block by behaviour pattern, not by source.
Blocking without having counted first. The most expensive mistake. A badly calibrated Block
rule on a Friday at 19:00 does more damage than the attack. Count first, always.
Confusing a legitimate spike with an attack. Look at the conversion to order and the cache hit rate before anything else. If the orders rise with the traffic, it is business, not an attack.
Switching off CloudFront "to rule it out". It removes the protection and exposes the origin. Never.
Tip: rehearse the plan. A quarterly half-hour drill —"the alarm goes off, what do you do?"— is
worth more than a perfect document nobody has read. And check that the notifications from
alertas-mercadofresco really do reach a phone, not just an inbox nobody looks at overnight.
Tip: keep a baseline. Have the normal traffic per hour and day of the week to hand. Without a baseline you cannot say whether 5,000 requests per second is a lot.
Tip: separate the expensive routes. Search, /login and /api/pedidos in different target
groups —tg-mercadofresco-api has existed since 03-03— lets you protect them with specific rules and
stops their saturation dragging the rest of the catalogue down with them.
Exercises
Exercise 1: deciding about Shield Advanced with numbers
A concert ticketing company, with an architecture identical to MercadoFresco's, turns over 40 million euros a year concentrated in the hours following each on-sale. An hour of downtime during an on-sale costs around 350,000 euros. Last year it suffered two layer 7 attacks that degraded the service for 40 minutes each, and it received an extortion email threatening to repeat them. It has no night-time on-call team.
Assess whether it should buy Shield Advanced. Structure the answer as: annual cost of the service, expected loss without it, specific features that add value in this case, additional requirements that have to be budgeted for, and the decision with its justification.
Exercise 2: characterising an incident
On a Tuesday at 04:12 MercadoFresco's alarms fire. The data for the first ten minutes:
- The ALB's
RequestCount: from 200/min to 45,000/min in 30 seconds. - CloudFront's
CacheHitRate: from 89 % to 11 %. - The ASG's
CPUUtilization: 97 % on both instances; the ASG scales to 12. - RDS's
DatabaseConnections: from 25 to 190 (the limit is 200). - Metric
MercadoFresco/Tienda/PedidosPorHora: 3 (normal at that hour is 5-10). - ALB logs: 8,400 distinct IPs, 61 countries, 78 % of them asking for
/buscar?q=<random string>&pagina=<random number>. - User agent: 91 % declare
Mozilla/5.0 (compatible; Baiduspider/2.0).
Answer: (a) attack or legitimate spike, and which three signals justify it?; (b) which layer and which technique?; (c) why is it especially effective against this architecture?; (d) which three measures would you apply, in order, and what is the risk of each?; (e) what would Shield Advanced have done that cannot be done without it?
Exercise 3: closing off direct access to the origin
Write the complete procedure, with commands, to stop anybody hitting alb-mercadofresco-tienda while
bypassing CloudFront. It must cover: where the secret value is stored, how CloudFront sends it, how
the ALB verifies it, what happens to requests that do not carry it, how it is further reinforced with
security groups, and how the value is rotated without causing an outage.
Solutions
Solution 1
Annual cost of the service:
| Item | Amount |
|---|---|
| Shield Advanced (3,000 USD × 12, annual commitment) | 36,000 USD |
| Business support (needed for the proactive SRT) | from 1,200 USD |
| Data transfer from protected resources | Variable, a few thousand |
| Approximate total | ~40,000 USD/year |
Expected loss without it: two 40-minute incidents last year × 350,000 €/hour × 0.67 h ≈ 470,000 € a year in direct loss, without counting reputation or the fact that there is an explicit threat to repeat, which raises the future probability.
Features that add value in this specific case:
- Automatic layer 7 protections. There is no night-time on-call and the attacks happen in the sales windows, which can be in the small hours. Shield Advanced writes and applies rules on its own, in seconds. It is the decisive feature here.
- SRT with proactive engagement. It literally makes up for the absence of an on-call team.
- Cost protection. With that concentration of traffic, the scaling during an attack can generate charges of tens of thousands.
- Per-resource detection. This company's traffic is extremely spiky by nature; a generic baseline would not tell an attack from an on-sale. Only the resource-specific baseline can.
Additional requirements to budget for: the Business or Enterprise support plan, the work of configuring Web ACLs and permissions for the SRT, defining emergency contacts and a rehearsal of the procedure.
Decision: buy it, without hesitation. 40,000 USD against an exposure of 470,000 € a year is a ratio of more than ten to one, with an explicit threat to repeat and no internal capacity for a night-time response. It is exactly the profile the product exists for. And it is worth underlining the contrast with MercadoFresco: the same service, the same architecture and the opposite decision, because the deciding variable is not technical but the cost of a minute of downtime.
Solution 2
(a) An attack, with three conclusive signals:
- Conversion has collapsed in relative terms. 45,000 requests per minute produce 3 orders an hour, when normal at that hour with 200 requests a minute is 5-10. If they were real customers, the orders would have risen with the traffic.
- The cache hit rate collapses from 89 % to 11 %. A legitimate spike asks for popular content and holds or improves it. Here unique URLs are being requested on purpose.
- The curve is vertical: from 200 to 45,000 in 30 seconds, and at 04:12 on a Tuesday, which is the absolute traffic trough of a Spanish grocery shop.
As additional confirmation: 8,400 IPs in 61 countries do not correspond to MercadoFresco's customer
geography, and 91 % of identical user agents declaring themselves a Chinese crawler is an obvious
forgery —real crawlers identify themselves and respect robots.txt, and none of them
generates 45,000 requests a minute against a search.
(b) Layer 7, an HTTP GET flood aimed at the search endpoint with random parameters. The volume in bits per second is modest; the damage does not come from the bandwidth.
(c) It is especially effective for three chained reasons that attack precisely the strong points of the architecture:
- It nullifies the CloudFront cache. Each
q=<random string>generates a different cache key, so 100 % of the requests are cache misses and reach the origin. The CDN, which is the first line of defence, is neutralised. - It attacks the most expensive route. Search is not cached, runs a complex query and touches the database on every request.
- It saturates the resource that does not scale. The ASG scales to 12 instances, but each
instance opens connections to
mercadofresco-pedidos, which has a limit of 200. The scaling makes the problem worse: more instances mean more connections against a database that does not grow. At 190 of 200 connections, the next step is the whole shop being unable to query anything.
(d) Three measures, in order:
- Rate-based WAF rule on
/buscar, inCountfor 2-3 minutes. Risk: none, it blocks nothing; it only costs those minutes. It serves to confirm how many legitimate requests would fall. - Move the rule to
Blockwith a per-IP threshold, and add a block on the spoofed user agent. Risk: low, because at 04:12 legitimate traffic is minimal and the pattern is very well characterised. A legitimate crawler getting blocked does no real damage at that hour. - Increase the minimum TTL of search responses and cache the errors. Risk: slightly stale
search results for a few hours, acceptable. As a fourth measure, if the database is still at its
limit, divert the reads to
mercadofresco-pedidos-lectura.
What you must not do: block the 8,400 IPs one by one —they will come back with others—, or turn off search entirely, or point the DNS at the origin.
(e) Shield Advanced would have done three things that are impossible without it: detect the
anomaly against this resource's specific baseline rather than against a fixed threshold;
automatically apply WAF rules at 04:12 without Marta waking up, which is exactly the window in
which it happened; and provide the DDoSDetected and DDoSAttackRequestsPerSecond metrics that
characterise the attack without having to download and analyse gigabytes of logs by hand. On top of
that, it would allow a credit request for the cost of scaling to 12 instances.
Solution 3
Complete procedure.
1. Generate and store the secret value. It goes to Secrets Manager (04-03), not to a file or the console:
aws secretsmanager create-secret \
--name "mercadofresco/produccion/cdn/cabecera-origen" \
--description "Value of the X-Origen-Verificado header between CloudFront and the ALB" \
--kms-key-id alias/mercadofresco-datos \
--secret-string "$(openssl rand -hex 32)" \
--tags Key=Proyecto,Value=mercadofresco Key=Componente,Value=cdn \
--profile mercadofresco-dev2. Have CloudFront send it. In the definition of the origen-tienda-alb origin of distribution
E2QWERTY123ABC you add a custom header X-Origen-Verificado with that value. It is sent on
every request to the origin and it is not visible to the client.
3. Have the ALB verify it. A priority 1 rule on the HTTPS listener that forwards to the target group only if the header matches:
aws elbv2 create-rule \
--listener-arn <listener-443-arn> \
--priority 1 \
--conditions '[{"Field":"http-header",
"HttpHeaderConfig":{"HttpHeaderName":"X-Origen-Verificado",
"Values":["<current-value>"]}}]' \
--actions '[{"Type":"forward","TargetGroupArn":"<tg-mercadofresco-tienda-arn>"}]' \
--profile mercadofresco-dev4. What happens to the rest. The listener's default action becomes a fixed 403 response, so that any request without the header —that is, any request not coming from CloudFront— is rejected at the balancer, without reaching the instances and without consuming anything:
aws elbv2 modify-listener --listener-arn <listener-443-arn> \
--default-actions '[{"Type":"fixed-response",
"FixedResponseConfig":{"StatusCode":"403","ContentType":"text/plain",
"MessageBody":"Direct access not allowed"}}]' \
--profile mercadofresco-dev5. Reinforcement with security groups. You restrict the inbound rule of sg-mercadofresco-alb on
port 443 to CloudFront's managed prefix list, instead of 0.0.0.0/0:
LIST=$(aws ec2 describe-managed-prefix-lists \
--filters "Name=prefix-list-name,Values=com.amazonaws.global.cloudfront.origin-facing" \
--query 'PrefixLists[0].PrefixListId' --output text --profile mercadofresco-dev)
aws ec2 authorize-security-group-ingress \
--group-id sg-mercadofresco-alb \
--ip-permissions "IpProtocol=tcp,FromPort=443,ToPort=443,PrefixListIds=[{PrefixListId=$LIST}]" \
--profile mercadofresco-dev
aws ec2 revoke-security-group-ingress \
--group-id sg-mercadofresco-alb --protocol tcp --port 443 --cidr 0.0.0.0/0 \
--profile mercadofresco-devThey are two independent layers: even if somebody worked out the header value, they would also have to originate the traffic from a CloudFront range.
6. Rotation without an outage. The key is to accept two values simultaneously during the
transition, exactly the same principle as AWSCURRENT/AWSPENDING from 04-03:
- Generate the new value and store it in the secret as the
AWSPENDINGversion. - Change the ALB rule so it accepts both values in the
Valuesarray. - Update the CloudFront origin with the new value and wait for the distribution to finish deploying to every point of presence (a few minutes).
- Verify in the logs that no request with the old value is arriving any more.
- Remove the old value from the ALB rule and promote the version to
AWSCURRENT.
Reversing the order —changing CloudFront before the ALB accepts the new value— causes a total service outage for the whole deployment of the distribution. It is the classic mistake in this operation, and the reason it is done on a Tuesday morning and not a Friday afternoon.
Conclusion
You now know what you are protecting yourself against. A distributed denial-of-service attack is
not out to steal anything: it is out to exhaust a finite resource, and that is why the defence is not
a permission or a key but capacity, distance and filtering. You understand why the DENY rule
numbered 50 from 03-02 worked against one IP and is useless against ten thousand: a NACL takes 40
entries, and filtering inside your VPC means the packet has already consumed your bandwidth.
Malicious traffic has to be dropped at the edge, far away.
You can tell the two families and their defences apart: the volumetric attacks at layers 3 and 4 —UDP flood, reflection and amplification, SYN flood, fragmentation—, measured in gigabits and absorbed by the AWS network; and the layer 7 attacks —HTTP flood, slowloris, search abuse, credential stuffing—, which arrive with ridiculous volume, look like legitimate traffic and are the ones that really threaten a shop like MercadoFresco. Shield for the first, WAF for the second.
You know that Shield Standard is already on and free, that it acts on CloudFront, Route 53 and
ELB, and that MercadoFresco has been using it since 03-04 without having decided to: putting a CDN in
front was a security decision too. And you know what Shield Advanced adds —per-resource
detection, automatic layer 7 protections, the DDoSDetected and DDoSAttackBitsPerSecond metrics,
cost protection against scaling and access to the 24/7 SRT— along with its real price of 3,000 USD
a month with an annual commitment. You have done the assessment with numbers and the answer for
MercadoFresco is no: 36,000 dollars a year to cover an infrastructure risk of two hundred. And
you know exactly what would have to change for the answer to be the opposite, because the ticketing
company exercise is the same architecture with the opposite decision.
What you have done is what genuinely protects a company of this size: architecture. Serving
everything from the edge, reducing the surface by closing off direct access to the ALB with a
verified header stored in Secrets Manager and CloudFront's prefix list on sg-mercadofresco-alb,
caching aggressively including the errors, and over-provisioning with a deliberate ceiling on
the ASG so the scaling does not become the attacker's bill. You have set up three alarms towards
alertas-mercadofresco for 0.30 USD a month, knowing the CloudFront one goes in us-east-1. And you
have a response plan with a clear order and, above all, the table that tells an attack from a
very good Friday: the conversion to order and the cache hit rate ahead of anything
else.
But the plan always ends at the same point. When Marta identifies the pattern —those routes, those
countries, that spoofed user agent, that request rate per IP— she needs a tool that looks inside
the HTTP request and decides based on its content, not on its source. Neither Shield nor a
security group can do that: one works with packets and the other with IPs and ports. In lesson
04-05, "AWS WAF", we will close the module with that tool: the Web ACL associated in front
of the distribution E2QWERTY123ABC and of alb-mercadofresco-tienda, the AWS managed rule
groups, the match statements and the text transformations that prevent trivial evasions, the
rate-based rules to protect /login and /api/pedidos, and —above all— the deployment method
that avoids disaster: start in Count, read the logs, adjust the exceptions and only then move
to Block.
AWS Course
Module 1: Introduction to AWS
- What Is AWS?
- Setting Up Your AWS Account
- AWS Global Infrastructure
- The AWS Management Console
- AWS CLI and SDKs
Module 2: Core AWS Services
Module 3: Networking and Content Delivery
Module 4: Security and Identity
- AWS Identity and Access Management (IAM)
- AWS Key Management Service (KMS)
- Secrets Manager and Parameter Store
- AWS Shield
- AWS WAF
Module 5: Monitoring and Management
Module 6: Databases
Module 7: Application Integration
- Amazon SQS
- Amazon SNS
- Amazon EventBridge
- AWS Step Functions
- Integration Patterns: Idempotency, Retries and Dead-Letter Queues
