We finished 04-02 with the mercadofresco-pedidos database encrypted with AES-256 and a KMS key with a policy, rotation and separation of duties. And even so, anybody who sits down at Luis's laptop can connect to that database, because the password of the mfadmin user is written in plaintext in at least five places:

  1. In the user data of the launch template lt-mercadofresco-tienda, which can be read from the metadata service of any shop instance.
  2. In /etc/mercadofresco/tienda.conf inside every instance.
  3. In a .env file that Luis sent to himself over chat when he set up the development environment.
  4. In the command history of the session in which he first used it.
  5. In Marta's head; she chose it fourteen months ago and it has not changed since.

Encryption does not help here. As we saw in the first table of 04-02, encryption at rest protects against the stolen disk and the snapshot shared by mistake, but it does not protect against valid credentials in the wrong hands. This lesson solves exactly that problem.

AWS offers two services for looking after configuration values and credentials: AWS Systems Manager Parameter Store and AWS Secrets Manager. They are not competitors, they are complementary, and half of this lesson is about knowing which one to use for what.

Warning. The examples are teaching material and the credentials are fictitious. Any real management of credentials, rotation and compliance (GDPR, PCI DSS) must be reviewed by a security professional before being applied to an environment with customer data. A badly migrated secret —especially a badly configured rotation— leaves the application out of service with no obvious way back. Never use real credentials in a test environment.

Contents

  1. Why credentials in code are a structural problem
  2. What exactly happens when a secret reaches Git
  3. Configuration and secrets are not the same thing
  4. Parameter Store: parameter types
  5. Hierarchy by paths, versions and tiers
  6. MercadoFresco's configuration in Parameter Store
  7. Secrets Manager: secrets, versions and stages
  8. Encryption with KMS and resource policies
  9. Automatic rotation: the four-step cycle
  10. Configuring rotation for mercadofresco/produccion/rds/mfadmin
  11. Single user versus two-user alternation
  12. Comparison table and the recommendation for MercadoFresco
  13. Consuming from the application: boto3 and cache
  14. Consuming from Lambda, EC2, ECS and CloudFormation
  15. Minimum permissions to read a secret
  16. The migration: getting the password out of the user data
  17. What not to store here
  18. Auditing access to secrets
  19. Detecting leaked secrets in the repository
  20. Cost, quotas and cleanup

Why credentials in code are a structural problem

It is not carelessness, it is a design problem. A credential written in a configuration file has four properties that condemn it:

Property Consequence
It gets copied Every deployment, every backup, every new laptop multiplies the copies
It does not expire The mfadmin password has been the same for fourteen months, and could have been for ten years
It leaves no trace Nobody knows who has read it; no record is possible
It cannot be revoked without breaking something Changing it means updating every place at once

The fourth one is what perpetuates the problem. Marta knows the password ought to be changed, but changing it means editing the launch template, restarting every instance in the ASG, warning Luis and crossing her fingers. Since the risk of the operation looks bigger than the risk of doing nothing, it never gets done. Automatic rotation breaks that circle, and that is why it is the heart of this lesson.

What exactly happens when a secret reaches Git

It is worth being concrete, because the most expensive mistake is believing that deleting it in the next commit is enough:

# Luis notices and "fixes it"
git rm --cached .env
git commit -m "Remove the .env from the repository"
git push

This deletes nothing. The file is still in the history, reachable with git show <previous-commit>:.env. And if the repository was on a shared server or somebody ran a clone, those copies have it too. The only correct procedure is:

  1. Rotate the credential immediately. Consider it compromised from the moment it was written. This step is mandatory and it is not optional.
  2. Rewrite the history (git filter-repo, BFG) and force the push. This breaks the clones of the whole team, so it has to be coordinated.
  3. Search the logs to see whether the credential was used from anywhere unexpected.
  4. Add a scanner that stops it happening again.

The order matters: rotate first. Rewriting the history while the credential is still valid is treating the symptom. If the repository is public, the bots that crawl GitHub find AWS credentials within minutes, and the cost of an account used to mine cryptocurrency is counted in thousands of dollars per day.

Configuration and secrets are not the same thing

This distinction decides which service to use:

Configuration Secret
Example Bucket name, page size, timeout Password, API key, token
Can it appear in a log? Yes, with no consequences Never
Does it need rotating? No Yes, periodically
Who can see it? The whole technical team Only whoever needs it
Service Parameter Store Secrets Manager

Borderline cases worth settling in advance:

  • The database user name (mfadmin) is configuration. The password is a secret. But they are stored together, because rotation changes both in the alternation case.
  • The RDS endpoint is configuration: it is not secret and is already public inside your VPC.
  • A payment provider's API key is a secret, even one from a test environment.
  • The CloudFront distribution identifier (E2QWERTY123ABC) is configuration.

Parameter Store: parameter types

Parameter Store is a component of AWS Systems Manager. It stores key-value pairs with three types:

Type Encrypted Use
String No Simple values: names, URLs, numbers
StringList No Comma-separated list: eu-west-1a,eu-west-1b
SecureString Yes, with KMS Low-profile sensitive values
# A simple parameter
aws ssm put-parameter \
  --name "/mercadofresco/produccion/tienda/nombre-bucket-fotos" \
  --value "mercadofresco-catalogo-fotos" \
  --type String \
  --description "Shop photo catalogue bucket" \
  --tags Key=Proyecto,Value=mercadofresco Key=Entorno,Value=produccion \
  --profile mercadofresco-dev

# A list
aws ssm put-parameter \
  --name "/mercadofresco/produccion/red/zonas" \
  --value "eu-west-1a,eu-west-1b" \
  --type StringList \
  --profile mercadofresco-dev

# A value encrypted with our key from 04-02
aws ssm put-parameter \
  --name "/mercadofresco/produccion/tienda/clave-api-mensajeria" \
  --value "fictitious-1234567890" \
  --type SecureString \
  --key-id alias/mercadofresco-datos \
  --profile mercadofresco-dev

Notes on SecureString:

  • If you do not give --key-id, the AWS managed key alias/aws/ssm is used, which is free but not controllable. For values that matter, use your own key.
  • To read the decrypted value you need --with-decryption and the kms:Decrypt permission on the key. It is the same double authorisation as SSE-KMS.
aws ssm get-parameter \
  --name "/mercadofresco/produccion/tienda/clave-api-mensajeria" \
  --with-decryption \
  --query 'Parameter.Value' --output text \
  --profile mercadofresco-dev

Hierarchy by paths, versions and tiers

The hierarchy by paths is the feature that makes Parameter Store useful. Names are structured as paths and can be read in bulk:

/mercadofresco/
├── produccion/
│   ├── tienda/
│   │   ├── nombre-bucket-fotos
│   │   ├── url-cdn
│   │   ├── pedidos-por-pagina
│   │   └── tiempo-espera-segundos
│   ├── basedatos/
│   │   ├── punto-enlace
│   │   ├── punto-enlace-lectura
│   │   ├── nombre-bd
│   │   └── usuario
│   └── red/
│       └── zonas
└── desarrollo/
    └── tienda/
        └── ...

With a single call the application loads its whole configuration:

aws ssm get-parameters-by-path \
  --path "/mercadofresco/produccion/tienda/" \
  --recursive --with-decryption \
  --query 'Parameters[].[Name,Value]' --output table \
  --profile mercadofresco-dev

And the hierarchy turns directly into access control: a policy granting ssm:GetParametersByPath on /mercadofresco/produccion/* leaves the entire development environment out without listing a single parameter.

Versions. Every put-parameter --overwrite creates a new version and keeps the previous ones. You can read a specific version with name:number, and label a version with an alias:

aws ssm put-parameter --name "/mercadofresco/produccion/tienda/pedidos-por-pagina" \
  --value "50" --type String --overwrite --profile mercadofresco-dev

aws ssm label-parameter-version \
  --name "/mercadofresco/produccion/tienda/pedidos-por-pagina" \
  --parameter-version 3 --labels estable --profile mercadofresco-dev

# Read a specific version or a label
aws ssm get-parameter --name "/mercadofresco/produccion/tienda/pedidos-por-pagina:2" \
  --profile mercadofresco-dev
aws ssm get-parameter --name "/mercadofresco/produccion/tienda/pedidos-por-pagina:estable" \
  --profile mercadofresco-dev

Rolling back after an unfortunate change is a matter of moving the estable label to the previous version. There is no deployment involved.

Tiers.

Standard Advanced
Parameters per account and region 10,000 100,000
Value size 4 KB 8 KB
Parameter policies (expiry, notifications) No Yes
Storage cost Free 0.05 USD per parameter per month
Cost of calls Free up to 40/s 0.05 USD per 10,000

MercadoFresco fits into the standard tier with room to spare: zero euros for all its configuration. The advanced tier is justified when you need more than 4 KB —a certificate, for example— or the expiry policies, which warn you when a parameter has gone too long without changing.

There is also a high throughput mode (--parameter-tier, with ssm:GetParameters at 3,000 requests per second) that is billed separately; you only need it if the application reads parameters in the critical path of every request, which it should not be doing: that is what the cache is for.

MercadoFresco's configuration in Parameter Store

create() {
  aws ssm put-parameter --name "$1" --value "$2" --type "${3:-String}" --overwrite \
    --tags Key=Proyecto,Value=mercadofresco Key=Entorno,Value=produccion \
           Key=Componente,Value=tienda Key=Propietario,Value=marta \
           Key=CentroCoste,Value=tecnologia \
    --profile mercadofresco-dev
}

create "/mercadofresco/produccion/basedatos/punto-enlace" \
       "mercadofresco-pedidos.abc123.eu-west-1.rds.amazonaws.com"
create "/mercadofresco/produccion/basedatos/punto-enlace-lectura" \
       "mercadofresco-pedidos-lectura.abc123.eu-west-1.rds.amazonaws.com"
create "/mercadofresco/produccion/basedatos/nombre-bd" "pedidos"
create "/mercadofresco/produccion/basedatos/usuario" "mfadmin"
create "/mercadofresco/produccion/tienda/nombre-bucket-fotos" "mercadofresco-catalogo-fotos"
create "/mercadofresco/produccion/tienda/url-cdn" "https://d111111abcdef8.cloudfront.net"
create "/mercadofresco/produccion/tienda/pedidos-por-pagina" "50"
create "/mercadofresco/produccion/tienda/tiempo-espera-segundos" "30"
create "/mercadofresco/produccion/tienda/tema-alertas" \
       "arn:aws:sns:eu-west-1:111122223333:alertas-mercadofresco"

Notice that the mfadmin user name is here and the password is not. The user name is configuration; the password goes to the other service. And look at the side effect: the user data of lt-mercadofresco-tienda, which until now was a list of embedded values, becomes a generic call that is identical in every environment.

Secrets Manager: secrets, versions and stages

A Secrets Manager secret is a value —normally a JSON— encrypted with KMS, with versions and a rotation mechanism. The piece to understand properly is the version stages.

Each version of a secret carries one or more staging labels:

Stage Meaning
AWSCURRENT The current version. It is the one get-secret-value returns by default
AWSPENDING Candidate version during a rotation in progress, not yet validated
AWSPREVIOUS The previous version, kept so you can roll back

These three labels are the mechanism that stops rotation from cutting the service. For a few seconds the old password (AWSCURRENT) and the new one (AWSPENDING) coexist, and only when the new one has been tested are the labels swapped. No application is ever left without a valid credential at any moment.

Creating MercadoFresco's secret:

aws secretsmanager create-secret \
  --name "mercadofresco/produccion/rds/mfadmin" \
  --description "Credentials of the mfadmin user of mercadofresco-pedidos" \
  --kms-key-id alias/mercadofresco-datos \
  --secret-string '{
    "engine": "postgres",
    "host": "mercadofresco-pedidos.abc123.eu-west-1.rds.amazonaws.com",
    "port": 5432,
    "dbname": "pedidos",
    "username": "mfadmin",
    "password": "FictitiousTemporaryPassword2026",
    "dbInstanceIdentifier": "mercadofresco-pedidos"
  }' \
  --tags Key=Proyecto,Value=mercadofresco Key=Entorno,Value=produccion \
         Key=Componente,Value=basedatos Key=Propietario,Value=marta \
         Key=CentroCoste,Value=tecnologia \
  --profile mercadofresco-dev

The structure of the JSON is not arbitrary: those field names (engine, host, port, dbname, username, password) are exactly the ones the AWS managed rotation function for PostgreSQL expects. If you change them, rotation will not work. It is one of the details that wastes the most time.

--kms-key-id alias/mercadofresco-datos connects this lesson with the previous one: the secret is encrypted with the key we built in 04-02, and is therefore subject to its policy. Anybody who cannot use the key will not be able to read the secret even with secretsmanager:GetSecretValue. Once again the double authorisation.

Encryption with KMS and resource policies

Like a bucket, a queue or a key, a secret accepts a resource policy. It serves two purposes: cross-account access and reinforcement with Deny.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SoloLaTiendaYMartaLeenEsteSecreto",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalArn": [
            "arn:aws:iam::111122223333:role/rol-mercadofresco-tienda",
            "arn:aws:iam::111122223333:user/marta",
            "arn:aws:iam::111122223333:role/rol-rotacion-mfadmin"
          ]
        }
      }
    }
  ]
}

It is a Deny with StringNotEquals: "deny reading to anybody who is not one of these three". Remember from 04-01 that a Deny always wins, so not even an identity policy with secretsmanager:* on * can get past it. It is the most robust way of shielding the most sensitive secret in the account, and the rotation role must be on the list or rotation will fail.

aws secretsmanager put-resource-policy \
  --secret-id mercadofresco/produccion/rds/mfadmin \
  --resource-policy file:///tmp/politica-secreto.json \
  --block-public-policy \
  --profile mercadofresco-dev

--block-public-policy rejects the policy if it would grant public access. Always use it.

Automatic rotation: the four-step cycle

Here is where Secrets Manager adds its distinctive value. Rotation is run by a Lambda function, which AWS provides ready-written for the usual database engines, and which is invoked four times with a different step each time.

sequenceDiagram
    participant SM as Secrets Manager
    participant L as Rotation Lambda<br/>SecretsManagerRDSPostgreSQL...
    participant DB as mercadofresco-pedidos
    participant A as Shop (application)

    Note over SM,A: Day 30: time to rotate
    SM->>L: Step 1 - createSecret
    L->>L: Generates a random password
    L->>SM: PutSecretValue(stage=AWSPENDING)
    Note over SM: AWSCURRENT = old<br/>AWSPENDING = new

    SM->>L: Step 2 - setSecret
    L->>DB: ALTER USER mfadmin PASSWORD 'new'
    Note over DB: The DB now accepts the NEW one

    SM->>L: Step 3 - testSecret
    L->>DB: Connect with AWSPENDING and run a SELECT
    DB-->>L: OK
    Note over L: If it fails here, it aborts<br/>and AWSCURRENT stays intact

    SM->>L: Step 4 - finishSecret
    L->>SM: Move AWSCURRENT to the new version
    Note over SM: new = AWSCURRENT<br/>old = AWSPREVIOUS

    A->>SM: GetSecretValue (next read)
    SM-->>A: new password

The four steps, with what matters about each one:

Step What it does What happens if it fails
createSecret Generates the new password and stores it as AWSPENDING. It does not touch the database Nothing changes; it is retried
setSecret Changes the password in the database Delicate moment: the DB may have the new one while the secret has not promoted it
testSecret Connects with the new one and runs a test query Rotation is aborted; AWSCURRENT is still the old one
finishSecret Moves the AWSCURRENT label to the new version It is retried

The most important practical consequence is the one almost nobody anticipates: between setSecret and finishSecret there is a window in which the database already has the new password but AWSCURRENT still returns the old one. If your application caches the secret for an hour and opens a new connection right inside that window, it will get an authentication error.

That is why the correct pattern in the application is:

  1. Cache the secret (one read per request is both very expensive and slow).
  2. On receiving an authentication error, invalidate the cache and retry once.

That one-line retry is what turns rotation into something transparent. Without it, every rotation produces a handful of errors in the logs.

Configuring rotation for mercadofresco/produccion/rds/mfadmin

The simplest approach is to let Secrets Manager create the Lambda from its managed template:

aws secretsmanager rotate-secret \
  --secret-id mercadofresco/produccion/rds/mfadmin \
  --rotation-lambda-arn arn:aws:lambda:eu-west-1:111122223333:function:rotacion-mfadmin \
  --rotation-rules '{"AutomaticallyAfterDays": 30, "Duration": "2h",
                     "ScheduleExpression": "cron(0 3 ? * TUE *)"}' \
  --profile mercadofresco-dev

The --rotation-rules parameters:

  • AutomaticallyAfterDays: 30: every 30 days.
  • ScheduleExpression: when exactly. Here, Tuesdays at 3 in the morning. It is deliberate: never a Thursday or a Friday, because of the order peak.
  • Duration: "2h": a two-hour window within which it may start.

The rotation Lambda needs three things that are constantly forgotten:

  1. Network access to the database. It must be in the vpc-mercadofresco VPC, in the subnets snet-mercadofresco-app-a/-b, and sg-mercadofresco-basedatos must accept 5432 from its security group. What we saw in 03-02 applies here: reference the source SG instead of a CIDR.
  2. Access to the Secrets Manager API. As it sits in private subnets and its way out to the internet goes through nat-mercadofresco-a, you either accept that cost or you create an interface VPC endpoint for secretsmanager, just as we did with vpce-mercadofresco-s3 in 03-01. Without one of the two, the Lambda hangs until the timeout expires and the symptom tells you nothing.
  3. Permissions. Its role needs secretsmanager:GetSecretValue, PutSecretValue, UpdateSecretVersionStage, DescribeSecret, GetRandomPassword, plus kms:Decrypt and kms:GenerateDataKey on alias/mercadofresco-datos.

Immediate manual rotation, very useful for testing before trusting the schedule:

aws secretsmanager rotate-secret \
  --secret-id mercadofresco/produccion/rds/mfadmin \
  --rotate-immediately \
  --profile mercadofresco-dev

# See the result
aws secretsmanager describe-secret \
  --secret-id mercadofresco/produccion/rds/mfadmin \
  --query '{Rotation:RotationEnabled, Last:LastRotatedDate, Next:NextRotationDate,
            Versions:VersionIdsToStages}' \
  --profile mercadofresco-dev

Test rotation in development before enabling it in production. It is the most important advice in this lesson: a badly configured rotation leaves the shop without a database, and a half-finished setSecret failure may require changing the password by hand to recover.

Single user versus two-user alternation

There are two rotation strategies, and the choice has real availability consequences:

Single user Two-user alternation
How it works Changes the password of the same user Alternates between mfadmin_a and mfadmin_b
Window without a valid credential It exists, of a few seconds It does not exist
Complexity Low Medium: two users to create and maintain
Open connections Keep working until they reconnect The same
When Applications with a correct retry Critical workloads with no margin for error

MercadoFresco starts with a single user because it is simpler and the shop implements the retry. When the volume grows —or when containers that start up and die constantly come into play, in module 10— two-user alternation will be the sensible option. The AWS managed template exists for both cases: SecretsManagerRDSPostgreSQLRotationSingleUser and ...RotationMultiUser.

Comparison table and the recommendation for MercadoFresco

Secrets Manager Parameter Store
Automatic rotation Yes, managed, with Lambda No
Cost 0.40 USD per secret per month + 0.05 USD/10,000 calls Free in the standard tier
Maximum size 64 KB 4 KB (8 KB advanced)
Encryption Always, with KMS Only SecureString
Resource policies Yes No
Cross-account access Yes Not directly
Hierarchy by paths No (but / is used by convention) Yes, with bulk reading
Versions Yes, with stages Yes, with labels
Integration with RDS Native, managed credentials No
Cross-region replica Yes, automatic No
Access from Parameter Store Yes: /aws/reference/secretsmanager/<name>
When to use it Credentials that must rotate Configuration and low-profile secrets

The recommendation for MercadoFresco, written up as a team rule:

Anything that is a credential capable of giving access to customer data goes to Secrets Manager, with rotation enabled. Everything else goes to Parameter Store.

Applied:

Value Where Why
mfadmin password Secrets Manager It gives access to personal data; it must rotate
Payment gateway API key Secrets Manager Third-party credential; it must rotate
mfadmin user name Parameter Store It is configuration
RDS endpoint Parameter Store It is not secret
Bucket names, CDN URL Parameter Store Pure configuration
Order messaging API key Parameter Store SecureString Sensitive but low impact, and no automatic rotation available

The cost of this split: 2 secrets × 0.40 = 0.80 USD a month, plus a few cents of calls. All the configuration, free. If we had put everything into Secrets Manager, the 25 values would cost 10 USD a month; if we had put everything into Parameter Store, we would have no rotation. The split is not purism, it is economics.

A very convenient practical detail: Parameter Store can read Secrets Manager secrets using the prefix /aws/reference/secretsmanager/. That way the application uses a single API for everything:

aws ssm get-parameter \
  --name "/aws/reference/secretsmanager/mercadofresco/produccion/rds/mfadmin" \
  --with-decryption --query 'Parameter.Value' --output text \
  --profile mercadofresco-dev

Consuming from the application: boto3 and cache

The naive version —reading the secret on every request— has three problems: latency of tens of milliseconds, cost per call and the risk of exceeding the request quota. The correct one uses a cache with invalidation on an authentication error:

import boto3
import json
import psycopg2
from botocore.exceptions import ClientError

SECRET = "mercadofresco/produccion/rds/mfadmin"

session = boto3.Session(region_name="eu-west-1")
sm = session.client("secretsmanager")

_cache = {"value": None}


def get_credentials(force_reload: bool = False) -> dict:
    """Returns the credentials, using the cache unless a reload is forced."""
    if _cache["value"] is None or force_reload:
        response = sm.get_secret_value(SecretId=SECRET)
        _cache["value"] = json.loads(response["SecretString"])
    return _cache["value"]


def connect():
    """Connects to the database and retries ONCE if the credential has rotated."""
    for attempt in (1, 2):
        credentials = get_credentials(force_reload=(attempt == 2))
        try:
            return psycopg2.connect(
                host=credentials["host"],
                port=credentials["port"],
                dbname=credentials["dbname"],
                user=credentials["username"],
                password=credentials["password"],
                connect_timeout=5,
            )
        except psycopg2.OperationalError as error:
            if "authentication" not in str(error).lower() or attempt == 2:
                raise
            # The password has rotated: the cache is invalidated and we retry

The for attempt in (1, 2) loop is all the logic you need: first attempt with the cache, second with a forced reload, and it only retries if the error is an authentication one. A network or timeout error propagates as it is, because reloading the secret would not fix it.

For production there is the official library, which adds time-based expiry and is more complete:

pip install aws-secretsmanager-caching
import boto3
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig

client = boto3.client("secretsmanager", region_name="eu-west-1")
cache = SecretCache(
    config=SecretCacheConfig(
        secret_refresh_interval=3600,   # refresh every hour
        max_cache_size=1024,
    ),
    client=client,
)

credentials = json.loads(cache.get_secret_string("mercadofresco/produccion/rds/mfadmin"))

With secret_refresh_interval=3600 you make 24 calls a day per instance instead of one per request: from tens of thousands down to two dozen.

Consuming from Lambda, EC2, ECS and CloudFormation

Lambda: the parameters and secrets extension. AWS publishes a layer that starts a small local HTTP server inside the execution environment and caches for you. You add the layer, configure it with environment variables and query it over localhost:

import os
import json
import urllib.request

PORT = os.environ.get("PARAMETERS_SECRETS_EXTENSION_HTTP_PORT", "2773")
TOKEN = os.environ["AWS_SESSION_TOKEN"]


def read_secret(name: str) -> dict:
    url = f"http://localhost:{PORT}/secretsmanager/get?secretId={name}"
    request = urllib.request.Request(url)
    request.add_header("X-Aws-Parameters-Secrets-Token", TOKEN)
    with urllib.request.urlopen(request) as response:
        return json.loads(json.loads(response.read())["SecretString"])


def handler(event, context):
    credentials = read_secret("mercadofresco/produccion/rds/mfadmin")
    ...

The extension keeps the cache between invocations that reuse the same execution environment —remember the cold and warm start model from 02-05—, so in practice a heavily invoked function makes one call every five minutes instead of one per invocation.

EC2. With the rol-mercadofresco-tienda role attached, boto3 finds the credentials on its own. There is nothing to configure other than the permissions.

ECS. Task definitions let you inject secrets directly as environment variables, without the code calling the API. We will see it in 10-01.

CloudFormation: dynamic references. You can reference a secret without writing it into the template:

Resources:
  BaseDatos:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceIdentifier: mercadofresco-pedidos
      MasterUsername: '{{resolve:ssm:/mercadofresco/produccion/basedatos/usuario}}'
      MasterUserPassword: '{{resolve:secretsmanager:mercadofresco/produccion/rds/mfadmin:SecretString:password}}'

The {{resolve:secretsmanager:<secret>:SecretString:<field>}} syntax is resolved at deployment time and the value never appears in the template or in the stack events. It is covered in more depth in 09-01.

Minimum permissions to read a secret

This is the policy attached to rol-mercadofresco-tienda:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LeerSoloElSecretoDeLaBaseDeDatos",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:mercadofresco/produccion/rds/mfadmin-??????"
    },
    {
      "Sid": "DescifrarConLaClaveDeDatos",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
        }
      }
    },
    {
      "Sid": "LeerLaConfiguracionDeProduccion",
      "Effect": "Allow",
      "Action": ["ssm:GetParameter", "ssm:GetParameters", "ssm:GetParametersByPath"],
      "Resource": "arn:aws:ssm:eu-west-1:111122223333:parameter/mercadofresco/produccion/*"
    }
  ]
}

Three details that matter:

  • The six question marks at the end of the ARN. Secrets Manager adds a random 6-character suffix to the ARN of every secret (...mfadmin-AbCdEf). If you write the ARN without the suffix, the policy matches nothing. -?????? uses the single-character wildcard to cover it. It is mistake number one with Secrets Manager and its symptom —AccessDenied with an apparently correct policy— baffles anybody. The alternative is to end in -*, which is somewhat laxer.
  • kms:ViaService limited to secretsmanager. The shop can decrypt through Secrets Manager and through S3 (thanks to the statement from 04-02), but it cannot call kms:Decrypt directly against any blob it gets hold of.
  • ssm:GetParametersByPath scoped to /mercadofresco/produccion/*. The production shop does not see the development configuration, and vice versa.

And what is not there: secretsmanager:ListSecrets, which would reveal the names of every secret in the account, nor access to any other secret. The shop reads exactly one. If the payment gateway secret is added tomorrow, granting it will take an explicit decision.

The migration: getting the password out of the user data

Initial state of the user data of lt-mercadofresco-tienda, as it was left in 02-01:

#!/bin/bash
cat > /etc/mercadofresco/tienda.conf <<'EOF'
DB_HOST=mercadofresco-pedidos.abc123.eu-west-1.rds.amazonaws.com
DB_USER=mfadmin
DB_PASSWORD=PlaintextPasswordEverybodyCanSee
EOF
systemctl start mercadofresco-tienda

The migration plan, in order and without cutting the service:

flowchart TD
    A["1. Create the secret with the<br/>CURRENT password"] --> B["2. Create the configuration<br/>parameters"]
    B --> C["3. Add permissions to<br/>rol-mercadofresco-tienda"]
    C --> D["4. Change the code:<br/>read from the secret with retry"]
    D --> E["5. New version of<br/>lt-mercadofresco-tienda WITHOUT the key"]
    E --> F["6. Progressive refresh<br/>of the ASG"]
    F --> G["7. Verify that everything works"]
    G --> H["8. Manual test rotation"]
    H --> I["9. Enable rotation every 30 days"]
    I --> J["10. Clean up: .env, history,<br/>configuration files"]

Step 1 is crucial and counter-intuitive: you store the current password, without changing it. That way steps 1 to 7 change nothing functionally and can be reverted at any moment. The password only changes at step 8, when everything else has already been tested.

New user data, without a single credential:

#!/bin/bash
set -euo pipefail

REGION=eu-west-1
BASE_PATH=/mercadofresco/produccion

# Non-sensitive configuration from Parameter Store
DB_HOST=$(aws ssm get-parameter --region $REGION \
  --name "$BASE_PATH/basedatos/punto-enlace" --query 'Parameter.Value' --output text)
DB_USER=$(aws ssm get-parameter --region $REGION \
  --name "$BASE_PATH/basedatos/usuario" --query 'Parameter.Value' --output text)

cat > /etc/mercadofresco/tienda.conf <<EOF
DB_HOST=$DB_HOST
DB_USER=$DB_USER
DB_SECRET=mercadofresco/produccion/rds/mfadmin
REGION=$REGION
EOF

systemctl start mercadofresco-tienda

The password is not written to any file: the user data leaves the name of the secret, and the application reads it into memory at start-up and whenever it needs it. The user data, which is visible from the instance metadata, no longer contains anything exploitable.

Deployment with a progressive refresh, as in 02-01:

aws ec2 create-launch-template-version \
  --launch-template-name lt-mercadofresco-tienda \
  --source-version '$Latest' \
  --launch-template-data "{\"UserData\":\"$(base64 -w0 /tmp/user-data-nuevo.sh)\"}" \
  --profile mercadofresco-dev

aws autoscaling start-instance-refresh \
  --auto-scaling-group-name asg-mercadofresco-tienda \
  --preferences '{"MinHealthyPercentage": 90, "InstanceWarmup": 120}' \
  --profile mercadofresco-dev

And finally step 10, the cleanup, which is the part everybody forgets:

  • Delete /etc/mercadofresco/tienda.conf from the old instances (which destroy themselves with the refresh).
  • Delete the .env from the laptops and rewrite the Git history if it ever got pushed.
  • Purge the command history of the sessions where it was used.
  • And, above all, rotate: the previous password must be considered compromised.

What not to store here

Neither Secrets Manager nor Parameter Store is a general-purpose store:

Do not store Why Where it goes
IAM access keys for a role They should not exist Roles and temporary credentials (04-01)
Files or binaries The 64 KB / 4 KB limit, and the cost S3 encrypted with KMS
Customer personal data It is not a database Encrypted RDS
Public TLS certificates There is a dedicated, free service ACM
Long-lived SSH private keys They should be ephemeral EC2 Instance Connect, Session Manager
Configuration that changes per request Latency and quotas In-memory cache or database

One case deserves a mention: for RDS there are also managed credentials (--manage-master-user-password), where RDS creates and rotates the secret automatically without any configuration on your part. It is the simplest option if the database is new; MercadoFresco does not use it because mercadofresco-pedidos already existed, but for a new one it is the first choice.

Auditing access to secrets

Every GetSecretValue call is recorded in CloudTrail (05-03), with who, when, from which IP and with what result. It is the property that no .env file can offer:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --start-time 2026-08-01T00:00:00Z \
  --query 'Events[].[EventTime,Username,CloudTrailEvent]' \
  --output text \
  --profile mercadofresco-dev

What Marta watches for:

  • Reads from unexpected principals —anybody who is not rol-mercadofresco-tienda or the rotation role.
  • Reads outside working hours or from unknown IPs.
  • A spike in reads, which suggests somebody is enumerating secrets.
  • DeleteSecret or PutResourcePolicy, which ought to be extremely rare.

In 05-01 we will turn this into a real CloudWatch alarm that notifies the alertas-mercadofresco topic; for now it is enough to know that the trail exists.

Detecting leaked secrets in the repository

Preventing is cheaper than rotating. Two layers:

On the laptop, before the commit. git-secrets installs a hook that rejects commits with credential patterns:

git secrets --install
git secrets --register-aws
# MercadoFresco's own patterns
git secrets --add 'mfadmin.*password'
git secrets --add --allowed 'FictitiousTemporaryPassword'   # exception for the documentation
git secrets --scan-history

--scan-history reviews the whole history, not just the current state. Running it for the first time in a repository with years behind it usually brings surprises.

In the pipeline. The local scan can be skipped; the pipeline one cannot. Tools such as gitleaks, trufflehog or the platform's own secret scanning run on every git push and stop the build. We will integrate it into the build phase in 08-02, together with the tests and the static analysis.

And one hygiene rule that is worth more than any tool:

.env
.env.*
*.pem
*.key
credentials
config/secrets.yml

Cost, quotas and cleanup

Item Price
Secrets Manager: per secret 0.40 USD a month (prorated by the hour)
Secrets Manager: API calls 0.05 USD per 10,000
Secrets Manager: replica in another region An additional 0.40 USD per region
Standard Parameter Store Free (10,000 parameters, 40 requests/s)
Advanced Parameter Store 0.05 USD per parameter per month + 0.05 USD/10,000 calls
Rotation (Lambda invocations) Normal Lambda pricing: cents a year
KMS Already accounted for in 04-02

Calculation for MercadoFresco:

Item Quantity Monthly cost
Secrets (mfadmin + payment gateway) 2 0.80 USD
Secrets Manager calls (6 instances × 24 reads/day × 30) ~4,320 0.02 USD
Parameters in standard Parameter Store 25 0.00 USD
Parameter Store calls (instance start-ups) ~2,000 0.00 USD
Rotation Lambda 1 invocation ×4 steps a month ~0.00 USD
Total 0.82 USD/month

Less than a dollar a month to remove the most common and most expensive risk in an infrastructure. Compare it with the cost of a notifiable breach under the GDPR and the discussion closes itself.

Cleanup. A secret is not deleted immediately: there is a recovery period of between 7 and 30 days, for the same reason as in KMS.

# Schedule the deletion with 30 days of recovery (recommended)
aws secretsmanager delete-secret \
  --secret-id mercadofresco/produccion/rds/mfadmin \
  --recovery-window-in-days 30 --profile mercadofresco-dev

# Having second thoughts
aws secretsmanager restore-secret \
  --secret-id mercadofresco/produccion/rds/mfadmin --profile mercadofresco-dev

# Immediate deletion: irreversible, tests only
aws secretsmanager delete-secret \
  --secret-id secreto-de-prueba --force-delete-without-recovery --profile mercadofresco-dev

# Parameters: immediate deletion, with no safety net
aws ssm delete-parameter --name "/mercadofresco/desarrollo/tienda/prueba" \
  --profile mercadofresco-dev

Watch out for the asymmetry: parameters are deleted instantly and cannot be recovered. Before deleting a whole Parameter Store path in bulk, export a copy.

Common Mistakes and Tips

Forgetting the six-character suffix in the secret's ARN. It is the most frequent mistake with Secrets Manager. The real ARN is ...secret:mercadofresco/produccion/rds/mfadmin-AbCdEf, and a policy that simply ends in mfadmin matches nothing. Use -?????? or -*.

Caching the secret without invalidating the cache on an authentication error. Every rotation then produces a batch of errors. The single retry is four lines of code and it solves it.

Putting the rotation Lambda outside the VPC. It will not be able to connect to mercadofresco-pedidos and rotation will fail at setSecret, which is the worst possible moment. It must be in the application subnets and sg-mercadofresco-basedatos must accept its security group.

Forgetting the route out to the Secrets Manager API from the private subnet. The Lambda hangs until the timeout expires and the error message says nothing useful. It needs NAT or an interface endpoint.

Changing the field names in the secret's JSON. The managed rotation function expects engine, host, port, dbname, username, password. If you rename them "so they read better", rotation does not work.

Enabling rotation in production without having tested it. Test it first on a copy of the database, and with --rotate-immediately so you do not wait 30 days to find out something is wrong.

Storing everything in Secrets Manager because it is "the safe place". 25 values are 10 USD a month instead of 0.80. Non-sensitive configuration goes to Parameter Store.

Scheduling rotation for a Thursday night. MercadoFresco's peak is on Friday. Rotate early on Tuesdays, when there are two working days ahead in which to react.

Tip: use a naming convention from day one. <project>/<environment>/<service>/<resource> lets you write policies by prefix and see at a glance which environment each thing belongs to.

Tip: store the endpoint inside the secret as well as in Parameter Store. It is what the managed rotation expects, and it saves the application from having to combine two sources to open a connection.

Tip: put an alarm on NextRotationDate. A rotation that has gone 45 days without running on a secret configured for 30 is a broken rotation that nobody has spotted.

Tip: audit every quarter who can read each secret. Access grows on its own, and list-secrets together with the resource policies gives you the picture in a minute.

Exercises

Exercise 1: splitting values between the two services

MercadoFresco integrates a payment gateway and a courier provider. These eight values appear:

  1. Production API key for the payment gateway (rotates every 90 days by provider policy).
  2. Base URL of the gateway's API: https://api.pasarela.example/v2.
  3. Merchant identifier: MF-2026-ES-0042.
  4. Signing secret for the gateway's incoming webhooks.
  5. Maximum gateway timeout: 8 seconds.
  6. User name and password for the courier provider's SFTP.
  7. List of postcodes with 24 h delivery.
  8. Password of a second, read-only reporting database.

Decide the service, the type and the full name for each one, following MercadoFresco's convention, and justify the three cases you consider most debatable. Work out the resulting monthly cost.

Exercise 2: diagnosing a rotation that fails

Marta enables rotation for mercadofresco/produccion/rds/mfadmin. The rotation runs and fails. The Lambda's logs show:

[ERROR] setSecret: Unable to connect to database with previous secret
of secret arn mercadofresco/produccion/rds/mfadmin-AbCdEf

And in Secrets Manager, describe-secret shows a version with the AWSPENDING stage that has been there for three hours, while AWSCURRENT still points at the old version.

Answer: (a) at which step of the cycle has it failed and what exactly does that message mean?; (b) list in order the four most likely causes and how to check each one; (c) is the application down right now, and why?; (d) what has to be cleaned up before retrying?

Exercise 3: writing the policy for the rotation role

Write the complete permissions policy for the role rol-rotacion-mfadmin, which the rotation Lambda assumes. It must be able to: read and write the secret mercadofresco/produccion/rds/mfadmin (and only that one), move version stages, generate random passwords, use alias/mercadofresco-datos through Secrets Manager, write its own logs and work inside vpc-mercadofresco.

Solutions

Solution 1

# Value Service Type Name
1 Payment API key Secrets Manager mercadofresco/produccion/pasarela/clave-api
2 Gateway URL Parameter Store String /mercadofresco/produccion/pasarela/url-base
3 Merchant identifier Parameter Store String /mercadofresco/produccion/pasarela/id-comercio
4 Webhook signing secret Secrets Manager mercadofresco/produccion/pasarela/secreto-webhook
5 Timeout Parameter Store String /mercadofresco/produccion/pasarela/tiempo-espera-segundos
6 SFTP credentials Secrets Manager mercadofresco/produccion/mensajeria/sftp
7 24 h postcodes Parameter Store StringList /mercadofresco/produccion/reparto/codigos-postales-24h
8 Reporting DB password Secrets Manager mercadofresco/produccion/rds/informes

Debatable cases:

  • The merchant identifier (3). It looks sensitive because it identifies the company to the provider, but on its own it authorises nothing: without the API key it is useless, and it appears on the invoices. It is configuration. If the provider treated it as a credential, the decision would change.
  • The webhook signing secret (4). It might look like configuration because it does not "open" anything: it is only used to verify incoming signatures. But anybody who knows it can forge payment confirmation notifications, in other words get free orders. It is a first-order secret.
  • The postcodes (7). They are not secret at all —they are published on the website— but they are configuration that changes often and is better kept out of the code so coverage can be extended without deploying. StringList with get-parameter and --query 'Parameter.Value' returns the comma-separated string.

Cost: 4 new secrets × 0.40 = 1.60 USD/month, plus the previous 2 = 2.40 USD/month in Secrets Manager. Parameter Store: 4 more parameters, 0 USD. If the eight values had all gone into Secrets Manager: 3.20 USD for these eight alone, four times more, with no advantage at all for the four that do not rotate.

Solution 2

(a) It failed at the setSecret step, the second of the cycle. The specific message —"unable to connect with previous secret"— says the Lambda tried to connect to the database with the current credential (AWSCURRENT) in order to run the ALTER USER, and could not. That is: the problem is not the new password, it is that the Lambda cannot authenticate with the old one.

(b) Four causes, in order of likelihood:

  1. Network. The Lambda is not in vpc-mercadofresco, or it is in the wrong subnets, or sg-mercadofresco-basedatos does not accept 5432 from its security group. Check: aws lambda get-function-configuration --query 'VpcConfig' and review the inbound rules of the database SG with describe-security-groups.
  2. The AWSCURRENT password is not the real one. If a made-up password was entered when the secret was created instead of the one mfadmin actually has, the Lambda cannot authenticate. Check: try connecting manually with psql using the AWSCURRENT value from a bastion host.
  3. The JSON fields are not the expected ones. If dbname, port or engine is missing, or if they are written differently, the Lambda builds the connection string wrongly. Check: read the secret and compare field by field with the expected format.
  4. The endpoint is wrong —for example, the old one was left behind after the migration to the encrypted instance in 04-02, which changed the DNS. Check: aws rds describe-db-instances --query 'DBInstances[].Endpoint.Address' and compare.

(c) No, the application is not down. And that is exactly the virtue of the stage design: AWSCURRENT still points at the old version, which is the one that still works in the database, and it is the one get-secret-value returns. The failed rotation is an urgent operational problem, but it is not a service incident. Had it failed after setSecret —with the password already changed in the database but AWSCURRENT not promoted— then there would be an outage.

(d) Before retrying you have to remove the version stuck in AWSPENDING, or Secrets Manager will try to continue the half-finished rotation instead of starting from scratch:

aws secretsmanager describe-secret --secret-id mercadofresco/produccion/rds/mfadmin \
  --query 'VersionIdsToStages' --profile mercadofresco-dev

aws secretsmanager update-secret-version-stage \
  --secret-id mercadofresco/produccion/rds/mfadmin \
  --version-stage AWSPENDING \
  --remove-from-version-id <pending-version-id> \
  --profile mercadofresco-dev

Then fix the cause, test in development and retry with --rotate-immediately.

Solution 3

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GestionarSoloEsteSecreto",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue",
        "secretsmanager:PutSecretValue",
        "secretsmanager:UpdateSecretVersionStage"
      ],
      "Resource": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:mercadofresco/produccion/rds/mfadmin-??????",
      "Condition": {
        "StringEquals": {
          "secretsmanager:resource/AllowRotationLambdaArn":
            "arn:aws:lambda:eu-west-1:111122223333:function:rotacion-mfadmin"
        }
      }
    },
    {
      "Sid": "GenerarContrasenasAleatorias",
      "Effect": "Allow",
      "Action": "secretsmanager:GetRandomPassword",
      "Resource": "*"
    },
    {
      "Sid": "UsarLaClaveSoloViaSecretsManager",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
        }
      }
    },
    {
      "Sid": "Registros",
      "Effect": "Allow",
      "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:eu-west-1:111122223333:log-group:/aws/lambda/rotacion-mfadmin:*"
    },
    {
      "Sid": "InterfazDeRedEnLaVPC",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "*"
    }
  ]
}

The reasoning behind each block:

  • The secret, one and only one, with the -?????? suffix and the condition secretsmanager:resource/AllowRotationLambdaArn, which makes sure this role can only operate on secrets whose rotation is assigned to precisely this Lambda. It is extra defence against somebody reusing the role.
  • GetRandomPassword does not accept a resource ARN; hence "Resource": "*". It is not dangerous: it generates random strings and gives access to nothing.
  • KMS scoped with kms:ViaService to Secrets Manager: the role cannot decrypt S3 objects with the same key.
  • The logs of the function's own log group, with the trailing :*.
  • The three ec2: actions are mandatory for any Lambda inside a VPC: it creates and destroys an elastic network interface on every cold start. They do not accept a specific ARN because the interface does not yet exist when the permission is checked. It is one of the very few times when "Resource": "*" is unavoidable, and it can be narrowed with conditions on ec2:Subnet and ec2:SecurityGroup if you want to push it further.

secretsmanager:ListSecrets, which would reveal the account's inventory of secrets, does not appear, nor does any RDS action: the Lambda changes the password over SQL, not through the AWS API.

Conclusion

The mfadmin password is no longer where it should not have been. It has left the user data of lt-mercadofresco-tienda, the instances' configuration file, Luis's .env and the command history, and it now lives in mercadofresco/produccion/rds/mfadmin, a secret encrypted with alias/mercadofresco-datos, protected by a resource policy that denies reading it to every principal that is not rol-mercadofresco-tienda, marta or the rotation role, and that rotates by itself every 30 days, early on Tuesdays, without anybody having to remember or take on the risk of doing it by hand.

You understand why rotation does not cut the service: the version stages mechanism —AWSCURRENT, AWSPENDING, AWSPREVIOUS— and the four-step cycle createSecretsetSecrettestSecretfinishSecret, where the new password is created, applied and tested before being promoted, and where a failure at testSecret aborts everything leaving the old one intact. You also know where the dangerous window is —between setSecret and finishSecret— and why the application must invalidate the cache and retry once on an authentication error: four lines that turn rotation into something invisible.

You can tell configuration from secret, and with it which of the two services to use. MercadoFresco's configuration lives in Parameter Store under /mercadofresco/produccion/..., with a hierarchy by paths that lets you load it all with one call and write permissions by prefix, with versions and labels for rolling back without deploying, and free in the standard tier. The secrets live in Secrets Manager, at 0.40 USD each. The lesson's total is 0.82 USD a month, and you can argue why putting everything in the expensive service would have cost ten times more without adding anything.

On the practical side you know how to read a secret from boto3 with a cache and invalidation, from Lambda with the parameters and secrets extension that caches between invocations, from EC2 with the role and no configuration at all, and from CloudFormation with {{resolve:secretsmanager:...}} dynamic references that never write the value into the template (09-01); in ECS they are injected in the task definition and we will see it in 10-01. You know the minimum permissions to read a secret —including the six-character ARN suffix, the mistake that wastes the most time— and you know the trail of every read is left in CloudTrail (05-03) and that leaked-secret scanning is automated in the pipeline (08-02).

MercadoFresco already has its house in order on the inside: minimal identities, encrypted data and rotated credentials. But all of that protects against threats that come through the API door. There remains the other half of the problem, the one that arrives through the front door and needs no credential at all. Remember the DENY rule numbered 50 with which in 03-02 we blocked that IP making thousands of requests per minute: it worked because it was one address. If tomorrow requests arrive from ten thousand different addresses spread all over the world, that rule is useless, the NACL has a hard limit on entries, and the Auto Scaling group will react exactly as it is designed to react —launching instances— turning the attack into a bill. In lesson 04-04, "AWS Shield", we will see what a distributed denial-of-service attack really is, how volumetric attacks differ from application-layer ones, what protection you already have enabled and free in CloudFront, Route 53 and the ALB, what Shield Advanced adds for 3,000 USD a month —and whether that makes sense for a company the size of MercadoFresco—, and how to tell an attack from a legitimate Friday afternoon peak in the heat of the moment.

© Copyright 2026. All rights reserved