The two previous lessons have always worked on one machine: meteo-01, with its RAID 1, its /dev/md0, its meteora.conf in mode 600 and its ext4 file system. We have virtualized inside it, we have contained processes inside it, but there was still a physical machine in a cabinet, that someone installed, that has a serial number and that leaves the service down if it breaks until somebody goes and fixes it.

This lesson removes that assumption. In the cloud, the machine stops being an object and becomes an API call: you ask for an instance and in forty seconds you have it; you delete it and it disappears; it dies on its own at three in the morning because the physical host failed and nobody warns you. That change is not merely administrative: it changes what administering an operating system means, it changes where the data must live, it changes what can be debugged and it changes how a service is designed so that it survives.

You are going to see what exactly changes and why, where the operating system's responsibility boundary lies in each service model, how an instance really boots and how it is provisioned with cloud-init, what immutable infrastructure and minimal operating systems are, how network storage behaves versus local SSD with concrete numbers, and how a container orchestrator is literally an operating system for the cluster, where the cgroups from the previous lesson and the scheduler from 02-02 reappear. We will finish by migrating Meteora, decision by decision, including what is not worth migrating. Mobile and real-time systems close the module in 06-04, and monitoring tools are already Module 7.

1. What changes when the machine stops being physical
2. Cattle, not pets
3. Service models and the operating system's boundary
4. Booting an instance, step by step
5. cloud-init: provisioning meteo-01 in the cloud
6. Instance metadata and its security risk
7. Immutable versus mutable infrastructure
8. Minimal and immutable operating systems
9. Cloud storage: block, ephemeral and objects
10. Software-defined networking
11. Orchestration: the cluster's operating system
12. Serverless functions, microVMs and sandboxes
13. Autoscaling and the cold start
14. Observability when the machine disappears
15. Cost and density as a design criterion
16. Final case study: migrating Meteora

Let us list the changes plainly, because each one has concrete technical consequences:

The truly disruptive row is expected lifetime. A physical server is designed to last; a cloud instance is designed on the assumption that it is going to die. Providers say so openly: the availability commitment for an individual instance is usually 99.5% monthly — some 3.6 hours of downtime a month — and the guarantees only go up when you spread the load across several zones.

From that comes this lesson's first design rule, which contradicts everything you learn administering a server: do not try to keep the machine from failing; design so that its failure does not matter.

The metaphor is from 2012 and is still the best explanation of the cultural shift:

• A pet has a proper name (meteo-01), is installed by hand, is fondly monitored, is patched with care and, when it gets sick, is nursed back to health. It is unique and irreplaceable.
• Cattle are numbered (meteo-api-7f3c), provisioned automatically and interchangeable. When one gets sick, it is not cured: it is replaced.

The practical consequences are uncomfortable at first and liberating afterwards:

• No manual changes. A hand-run apt install on an instance creates state no other instance has and that will be lost when it is replaced. If the change matters, it goes into the image or into the automation.
• No important data lives on the instance's disk. Everything that must survive moves off it: to a persistent volume, to a managed database, to object storage.
• SSH access stops being the main tool. Not because it is forbidden, but because if you need it to operate, something is missing from the automation.
• Reinstalling is the default answer. What in 05-04 was the outcome of a serious incident — rebuilding from scratch — is here a Tuesday-morning operation.

An honest caveat, because the dogma gets exaggerated: not everything can be cattle. A stateful database, a domain controller or a system with hardware-tied licenses are still pets, and pretending otherwise causes disasters. The useful rule is that state should be concentrated in as few pieces as possible, and that everything else should be interchangeable.

The relevant question for this course is not "what is the cloud", but who administers the operating system in each model:

The important thing is to understand what you gain and what you lose as you climb that ladder. You gain operations: fewer patches to apply, less configuration to maintain, fewer on-call shifts. You lose control and, very specifically, you lose this course's tools: on a PaaS you cannot look at /proc, you cannot tune swappiness, you cannot add an nftables rule, you cannot run strace. When something goes wrong and the provider does not give you the metric you need, you are blind.

And a warning that is not often stated: on IaaS you are still responsible for the entire operating system. It is the shared responsibility model. The provider guarantees that the hardware and the hypervisor work; kernel patches, the hardening from 05-03, the firewall, the users and the logs are still yours. A freshly created instance from a distribution's official image is exactly as insecure as a freshly installed server.

It is worth seeing the complete sequence, because it explains where each piece fits:

The pieces, one by one:

1. The machine image. A disk template with a system already installed — the same idea as the templates from 06-01 — identified by an ID. It can be the provider's official one, the distribution's, or your own, built by you with everything you need already inside. That last option is the basis of immutable infrastructure.
2. The instance type. It determines vCPUs, memory, network and sometimes local disk. Here you recover all of 06-01: those vCPUs are threads of a process on a shared host, and their steal time is real.
3. The root volume. It is created by copying (or linking) the image. When you are done, it is normally deleted with the instance, unless marked otherwise.
4. The network: a virtual interface in the software-defined network, with a private IP, and an associated security group.
5. The boot: firmware, boot loader, kernel, init. Exactly what happens on any machine; you will see it in detail in Services, Boot and systemd.
6. cloud-init: the piece that turns a generic image into your server.

cloud-init is a set of services that run on first boot, query the metadata service and apply the configuration you passed it, called user-data. It is a de facto standard: it works the same way on nearly every provider and also on local KVM, which makes it the natural tool for the test replica we created in 06-01.

What it does and the reason for each decision:

• #cloud-config on the first line is mandatory: it is the signature that tells cloud-init to interpret the YAML. Without it the file is silently ignored, and that is mistake number one.
• Users with a public key and no password. lock_passwd: true for the service account and ssh_authorized_keys for the operations one apply 05-02 directly. The sudo rule is scoped to one specific command, not to ALL.
• An explicit uid: 990, for the same reason as in the Dockerfile in 06-02: volumes are shared by number, not by name. If cloud-init assigns UID 999 and the volume has files owned by 990, you get broken permissions.
• mounts with noatime,nosuid,nodev: the hardening options from 04-03 and 05-03, declared in the provisioning instead of hand-edited into /etc/fstab. Here is the whole cultural shift: the configuration is declared, not applied.
• Configuration with no secrets. This is the critical point: the user-data is readable from inside the instance by any process, as we will see in a moment. Putting a password there is equivalent to publishing it. Secrets are read at startup from a secrets manager, authenticating with the instance's identity.
• runcmd at the end, in order, for what the declarative modules do not cover.

Every instance can query information about itself at a special local address, typically 169.254.169.254:

What you have just seen, and why it is the cloud's biggest specific security risk. That endpoint returns, with no authentication whatsoever, the temporary credentials of the role assigned to the instance. Any process on the machine can ask for them and act in the cloud with those permissions. And there is the problem: if your application has a server-side request forgery (SSRF) vulnerability — something as silly as a ?url= parameter the application downloads — an attacker can ask your own application to query 169.254.169.254 and hand back the credentials. They do not need to run code: it is enough that your server makes an HTTP request on their behalf.

It is exactly the pattern of the 2019 Capital One breach, which exposed the data of more than a hundred million people. The mitigations are concrete and all of them necessary:

• Require the token-based version of the metadata service (IMDSv2 and equivalents), which forces a prior PUT to obtain a session token with a low TTL: a simple SSRF, which only knows how to do GET, stops working.
• Block access to the endpoint from the processes that do not need it, with nftables or the container itself.
• A role with minimal permissions: if meteo-api only has to write to one specific bucket, the role should allow nothing else. It is least privilege from 05-01 applied to the cloud plane.
• Never put secrets in the user-data, because it is readable from there.

Two opposing philosophies for managing a server's lifecycle:

Configuration drift is the decisive argument. In a fleet of twenty mutable servers managed over two years, no machine is identical to another: an emergency apt install here, a hand-edited file there, a package that failed to update on two of them. The result is that a bug appears on only three servers and nobody knows why, and that "it works on mine" becomes the standard reply.

With immutable infrastructure, every instance of version 1.4.0 is bit-for-bit identical, so a bug is either on all of them or on none. And rolling back is redeploying 1.3.0, which turns a risky deployment into a boring operation.

What you lose, and it is real: debugging gets harder. If an instance misbehaves and the policy is to replace it, the problem disappears with it and you learn nothing. The correct practice is to take the sick instance out of the load balancer but not destroy it, leaving it in quarantine to be investigated — which connects directly with the evidence preservation from 05-04 — and to replace it with another to restore service. Containing and analyzing are not incompatible: they are different phases.

If the instance is cattle and only runs containers, why would it want a complete operating system with a compiler, a package manager, a mail server and fifty utilities?

What you gain is measurable and substantial:

• A tiny attack surface. With no shell there is no reverse shell; with no curl or wget the second stage of an attack cannot be downloaded; with no compiler a local exploit cannot be compiled. It is the minimal-surface principle from 05-01 taken to the extreme, and it eliminates entire classes of attack, it does not merely make them harder.
• Fewer patches. A system with 80 packages has fewer CVEs per month than one with 500. Less noise, fewer on-call shifts.
• Atomic, reversible updates. The A/B scheme writes the new image to the inactive partition and only switches the boot at the end: either it updates completely or it does not update, and if it does not boot properly, it goes back to the previous one by itself. It is the opposite of an apt upgrade that stops halfway.
• Fast boot, because there are very few services to start.

What you lose is equally real: debugging is awkward. With no strace, no tcpdump, no ss and sometimes no shell, Module 7's techniques cannot be applied directly. The ecosystem's answer is ephemeral debug containers: a container with the tools is launched inside the problematic process's namespaces — the nsenter from 06-02, industrialized — you investigate, and you throw it away. It works well, but you have to know how to do it before the incident happens, not during.

This is where Module 2 becomes very visible, and where the most design mistakes are made.

They behave like a disk: you partition them, format them with ext4 and mount them. But they are on the other side of a network, and that shows exactly where 02-05 predicts:

The key difference is not just the number: it is that IOPS are provisioned and paid for. A basic volume often gives 3 IOPS per GB, so a 100 GB volume delivers 300 IOPS — less than a spinning hard disk from 2005 — no matter how flash-based the underlying hardware is. It is the most common source of surprise when migrating: the same application, at the same version, runs four times slower, and the reason is not the CPU but a contracted IOPS ceiling.

For Meteora, the ingestor writes 8,000 readings per second. If it writes them one at a time, that is 8,000 IOPS and the basic volume chokes; if it batches them into groups of 512 readings (12,288 bytes), they drop to about 16 writes per second. Batching, which on the physical server was a minor optimization, is here the difference between working and not working.

It is the host's physical NVMe: extremely fast and often included in the instance's price. But it is lost when the instance stops or if the host fails. Its correct use is everything that can be rebuilt: caches, temporary files, derived indexes, intermediate data for a batch process. Never the source of truth.

It is the most misunderstood piece, because it is not a file system even though its interface looks like one:

That is why mounting it as if it were a disk is almost always a bad idea: the tools that translate POSIX into objects have to emulate what does not exist, and an mv of a large file turns into copying the whole object and deleting it, a readdir turns into several paginated HTTP requests, and there is no locking and no partial writing. It works in demos and it breaks in production with real data.

Applied to Meteora, the decision is clean:

One detail that rounds out the decision: object storage offers access classes with very different prices. A 17.3 MB daily file compresses down to about 4 MB; over five years that is about 7.3 GB, at a ridiculous cold-storage cost. Keeping that same data on block volumes costs an order of magnitude more and adds nothing, because nobody queries 14 March 2023 with millisecond latency.

The cloud's network is a simulation built on top of the provider's physical network, and its pieces map one to one onto what you already know:

• A private virtual network: your own address space (for example 10.0.0.0/16) isolated from other customers', subdivided into subnets per availability zone. Conceptually it is the bridge from 06-01, at data-center scale.
• Public and private subnets: the real difference is whether their route table has a path out to an Internet gateway. Databases and internal services go in private ones; only the load balancer lives in the public one.
• Security groups: a stateful firewall applied per network interface, not per machine. That is the difference from nftables: the rule travels with the instance, it is evaluated before the packet reaches the operating system, and by default it denies everything inbound.
• Addresses: the private IP is stable for the instance's lifetime; the public one, unless reserved, changes on every boot, which is why you must never configure anything by public IP.

Two pieces of advice that avoid most cloud networking incidents:

• The security group does not replace the system firewall. It is defense in depth, as in 05-03: if somebody gets the group's rule wrong, nftables with policy drop is still there.
• Reference groups, not IP ranges. The correct rule for the database is not "allow 10.0.1.0/24", but "allow traffic coming from meteo-api's security group". That way the rule stays correct when autoscaling creates instances with new IPs.

Here is the idea that makes this lesson belong by full right in an operating systems course. A container orchestrator does to a cluster exactly what an operating system does to a machine:

The component that closes the circle with the previous lesson is the kubelet, the agent running on every node. Its job is simple to describe: it asks the control plane which pods are its responsibility, talks to the container runtime (containerd or CRI-O, via the CRI interface) to start them, mounts their volumes, runs their health probes and reports status. It is, literally, the node's init for the cluster's workloads.

When you write this in a manifest:

The kubelet translates it exactly into the files we wrote by hand in 06-02:

And from that follow, with no magic at all, the two behaviors that most bewilder newcomers:

• Exceeding the CPU limit does not kill: it throttles. It is the cpu.max throttling we diagnosed in 06-02 with nr_throttled. The application does not fail, it just runs in fits and starts with a horrible p99.
• Exceeding the memory limit kills instantly. It is the cgroup's OOM killer: the container dies with code 137 and the pod shows up as OOMKilled. Exactly the aggregator case from the previous exercise.

The cluster scheduler is the equivalent of the CPU scheduler from 02-02, one level up: it filters out the nodes that will not do (not enough free resources according to the requests, missing the requested labels, with active constraints) and scores the ones left in order to spread the load. And here is an operational rule learned the hard way: the scheduler places according to requests, not according to actual use. If you request 2 CPUs and use 0.1, the cluster reserves 2 for you and places nobody else in that gap. Inflated requests are the number one cause of hugely expensive clusters with nodes at 15% utilization.

Serverless functions take the ladder to its extreme: you upload a function, the provider runs it when an event arrives and charges you per millisecond. There is no server to administer, no operating system to patch.

But underneath there is something, and that something is interesting for this course, because it poses a hard problem: code from thousands of different customers has to run on the same hardware, with virtual-machine isolation but container startup. The virtual machines from 06-01 isolate well but take tens of seconds; the containers from 06-02 start in milliseconds but share a kernel. Two technologies occupy that middle ground:

• Firecracker is a minimalist VMM written in Rust on top of KVM. It gives up almost everything QEMU emulates: no legacy BIOS, no USB, no VGA, no full PCI; only virtio for disk, network and console. The result is a microVM that boots in ~125 ms, consumes about 5 MB of overhead and exposes a tiny surface to the hypervisor. It is the real hypervisor behind serverless function platforms, and it is the practical demonstration of the VENOM lesson from 06-01: removing emulated devices improves security and performance at the same time.
• gVisor attacks the problem from the other side: it is a user-space kernel that intercepts the container's system calls and implements them itself, in Go, instead of letting them through to the host kernel. It drastically reduces the surface — the container no longer talks to Linux's 350 syscalls, but to gVisor — at the cost of a noticeable performance penalty on I/O-heavy workloads.

The conclusion worth keeping: the boundary between VM and container has stopped being binary. Firecracker achieves VM isolation with almost container-like startup by trimming everything that is not needed, and it is the technical reason serverless functions can be both multi-tenant and cheap.

Autoscaling adjusts the number of instances or pods according to a metric: CPU usage, queue length, requests per second. It sounds trivial and it has two real problems.

The first is the cold start. Scaling is not instantaneous, and the latency accumulates layer by layer:

If your load doubles in 30 seconds and a new instance takes 90 to be ready, autoscaling arrives late and users see errors. The answers are to scale on a leading metric (the ingestor's queue length grows before the CPU saturates), to keep a cushion of idle capacity, to preload images onto the nodes, and to scale aggressively up and conservatively down so as not to oscillate.

The second problem is oscillation: if the metric rises and falls around the threshold, the system creates and destroys instances endlessly, at high cost and with no stability. It is solved with hysteresis — different thresholds for scaling up and down — and with cooldown periods.

And an important nuance for Meteora: the ingestor does not scale the same way as meteo-api. HTTP queries are stateless and can be spread at will; ingesting from specific stations has affinity and ordering. Scaling something stateful horizontally without thinking leads to duplicated or lost readings.

Here you see starkly why 05-04 insisted so much on getting the logs off the machine. In the cloud, that recommendation goes from good practice to absolute requirement, for a new reason: the instance that generated the log may not exist when you go to read it.

• A pod that dies from OOM restarts and its previous logs disappear unless they have been shipped out.
• An instance replaced by autoscaling takes its journalctl with it.
• The ephemeral disk, with whatever was in /var/log, is wiped when the instance stops.

Hence the practical rules:

• The application writes to standard output, not to a file. An agent on the node collects, labels and ships. It is the opposite of /var/log/meteora/meteo-api.log, and in a container it makes total sense: the one who has to worry about where the log ends up is not the application.
• Structured logging with cloud context: to the request_id from 05-04 you now add instance identifier, zone, image version and pod name. Without that, an error occurring on one instance out of twenty is indistinguishable from noise.
• Distributed tracing, because a request crosses a load balancer, several services and a database: one log per service no longer reconstructs the story.
• Metrics in aggregate, not per machine. "meteo-01's CPU" stops meaning anything when there are fifteen ephemeral instances; what matters is the percentile and the distribution.

What do you lose when debugging? Quite a lot, and it is worth saying: you cannot go and look at /proc on an instance that no longer exists; you cannot run strace on a process that died ten minutes ago; you cannot reproduce the exact state of a wiped ephemeral disk. The compensation is that you can keep the sick instance in quarantine instead of killing it, which is the technique you need to have written into the procedure before you need it.

On a physical server, cost is an investment made two years ago and it does not influence day-to-day design. In the cloud, every technical decision has a monthly bill, and that makes cost an architectural criterion in its own right.

Some relationships worth internalizing:

• Oversizing is expensive and invisible. Inflated requests in a manifest produce no errors: they produce nodes at 15% utilization and a triple bill. Measuring actual use and adjusting is an engineering task, not an accounting one.
• Data transfer is charged for, and outbound especially. Moving the daily files between zones or out to the Internet can cost more than storing them. Placing compute next to the data stops being a latency question and becomes a money one.
• Storage classes matter a lot. We saw it already: closed days in cold storage cost a fraction of what they cost on block volumes.
• Interruptible (spot) instances cost between 60% and 90% less in exchange for being taken away from you with two minutes' notice. For the aggregator, which is a resumable batch job, they are ideal. For meteo-api, they are not.
• Density is savings. Every extra container that fits on a node is one fewer node to pay for; this is where a container's baseline cost of 1-10 MB versus a VM's 200-500 MB turns into euros.

Decision by decision, with the justification made explicit:

And what is not worth migrating, which is equally important:

• The complete historical storage, exactly as it is. Moving five years of daily files to block volumes would be extremely expensive and absurd: they go into objects, compressed and in a cold class. Migrating "just as it was" is the classic mistake.
• The /dev/shm/meteora-cache cache as a shared piece. On one machine it made sense; with replicas across several zones, a local per-instance cache is incoherent. Either you accept that each replica has its own (if it is rebuildable and tolerant of being stale), or you use a networked cache.
• The hand-written cron and maintenance scripts. They do not survive ephemeral instances: they become the orchestrator's scheduled jobs.
• The /run/meteora/readings.fifo FIFO between services. It is local IPC (03-03) and only works within one machine; between instances you need a real queue, with persistence and retries.
• Regulated data outside its jurisdiction. If there are data residency obligations, the region is not a technical decision but a legal one, with the implications from 05-04.
• And perhaps nothing at all, if the case does not justify it. A stable service, with predictable load and no need for elasticity, may end up more expensive and more fragile in the cloud. The cloud is paid for with money and with complexity; both must be justified.

Treating instances as pets. Installing by hand, configuring over SSH and trusting that the machine will still be there. The first automatic replacement wipes out all those changes and nobody knows how to rebuild them.

Putting secrets in the user-data. It is readable from inside the instance by any process through the metadata service. A secret there is a published secret.

Not protecting the metadata service. An SSRF in the application becomes theft of the instance role's credentials; it is the exact pattern of one of the largest known breaches. Require the token-based version, block access from where it is not needed and reduce the role's permissions.

Assuming a block volume performs like a local SSD. It is 5-10 times worse in latency and has an IOPS ceiling that you contracted. Design the application to batch writes before migrating, not afterwards.

Mounting object storage as if it were a disk. There is no partial writing, no locking, no fsync, no cheap renaming, and the latency is a thousand times higher. Use it with its API.

Confusing requests with limits. Requests decide where you are placed and what is reserved for you; limits decide when you are throttled or killed. Inflated requests give you hugely expensive clusters at 15% utilization; memory limits set too low give you restarts with code 137 and no trace at all.

Forgetting that autoscaling takes time. If your peak arrives in 30 seconds and the instance takes 90 to start, you have scaled late. Scale on a leading metric and keep a cushion.

Relying only on local logs. In an ephemeral environment, a log that has not left the machine is a log that has been lost.

Tip: start with the state. Before migrating anything, take an inventory of where each piece of data lives and what its source of truth is. Everything stateless migrates easily; everything else is where the hard decisions are.

Tip: try killing an instance on purpose. In a test environment, destroy a meteo-api replica during working hours and watch what happens. If the service does not recover by itself in under a minute, your design still treats that instance as a pet. It is the only way to find out before it happens for real.

Tip: set a cost alarm on day one. A misconfigured scaling loop, a forgotten volume or a cross-region transfer can multiply the bill without generating a single error.

Write a complete #cloud-config file that provisions an instance to run the aggregator, satisfying: (a) a meteora service user with UID 990, no shell and a locked password; (b) an operations account with a public key and sudo limited to restarting the service; (c) an additional disk formatted as ext4 and mounted at /var/lib/meteora with the course's options; (d) a firewall with a deny policy and SSH only from the private subnet; (e) the file /etc/meteora/meteora.conf in mode 600 with no secrets, explaining where they come from; (f) automatic security updates. Justify each block and point out three things you deliberately do not put there and why.

meteo-api is deployed with this manifest on nodes with 4 vCPUs and 8 GiB:

Measured in production, each replica uses a sustained 0.2 vCPU and 300 MiB, with peaks of 0.8 vCPU and 450 MiB.

(a) How many replicas fit per node with this manifest, and how many would fit with adjusted requests? Compute the waste and its impact on the number of nodes needed for 30 replicas. (b) Propose justified requests and limits values. (c) After the change, some pods start restarting with code 137 and others show a very high p99 latency without restarting: explain what is happening to each group, which cgroup file confirms it and how you would fix it. (d) Explain why setting requests equal to limits for memory is a good idea but for CPU usually is not.

Meteora generates a daily file of 17,280,000 bytes (720,000 readings × 24 B). The queries are: 92% over the last 48 hours, 7% over the last month and 1% over the five-year history. The ingestor receives 8,000 readings per second. The basic block volume offers 3 IOPS per GB.

(a) Compute the annual volume of raw and compressed data (4:1 factor), and design where each data set lives with its storage class. (b) Compute the IOPS the ingestor needs writing reading by reading and batching in groups of 512, and say what size of basic volume would be needed in each case just to reach those IOPS. (c) Justify why the current day's file cannot be in object storage, with at least three concrete technical reasons. (d) Design the backup policy satisfying the 3-2-1 rule from 05-03 and explain what immutability adds against ransomware.

Solution 1

Justification. The explicit UID 990 is essential because the volume is shared by number, not by name (the same reason as in the Dockerfile in 06-02). lock_passwd: true on both accounts implements the "public key only" rule from 05-02, and the sudo rule scoped to one specific command applies the least privilege from 05-01 — remembering its limit: it bounds what is executed, not what can be achieved. The noatime,nosuid,nodev mount options are the hardening from 04-03 declared, not hand-edited. The firewall with policy drop and SSH restricted to the private subnet is defense in depth in addition to the security group. And automatic updates close out the first measure from 05-03.

Three things that deliberately do not go there:

1. No secrets, because the user-data is readable by any process on the instance via 169.254.169.254/latest/user-data: putting one there is publishing it. It is read from the secrets manager at startup, authenticating with the instance's role.
2. No SSH private key and no TLS certificate, for the same reason. Only the operator's public key.
3. No heavy application installation (compiling, downloading large artifacts, configuring by hand). That goes into the image, built and versioned, not into the boot: if it goes here, every instance takes minutes to be ready, depends on the repositories responding and stops being reproducible — it is mutable infrastructure in disguise.

Solution 2

(a) Density calculation. With requests: 2000m / 4Gi on nodes with 4 vCPUs and 8 GiB: by CPU 2 pods fit, by memory 2 fit (and leaving headroom for the node's own system, in reality 1). Let us take 2 pods per node in the best case: for 30 replicas you need 15 nodes.

With requests adjusted to real use (for example 300m / 512Mi): by CPU 13 pods fit, by memory 16; CPU is the limit, about 12 pods per node leaving headroom for the system. For 30 replicas, 3 nodes are enough.

The waste is a factor of 5: 2000m is reserved to use 200m, that is, 10% of what is reserved is actually used. And since the scheduler places according to requests and not according to use, the cluster will show nodes at 5-10% real CPU and still be "full", unable to accept more pods. It is the number one cause of disproportionate bills.

(b) Proposed values:

Justification: the CPU request sits slightly above the sustained use (0.2 vCPU) so the scheduler reserves what is needed without inflating; the CPU limit is set generously (1 vCPU) to absorb the 0.8 peaks without throttling. For memory, request and limit are both 512 MiB, with headroom over the 450 MiB peak, because memory is not compressible.

(c) The two symptoms and their cause:

• The pods that restart with code 137 are being killed by the cgroup's OOM killer: their real consumption exceeds memory.max. The 137 is 128 + 9 (SIGKILL) and the pod's status will be OOMKilled. It is confirmed with memory.events (a non-zero oom_kill field) or with the pod's event. Fix: raise the memory limit above the real measured peak — perhaps the 450 MiB peak did not cover every case — and add an equivalent memory.high so there is pressure before death.
• The pods with a high p99 that do not restart are CPU-throttled: they have exhausted their cpu.max quota within the period and stay stopped until the next one. It is confirmed with nr_throttled versus nr_periods in cpu.stat, exactly as in 06-02. Fix: raise the CPU limit or adjust the application's thread count to the quota; if all that was wanted was priority, remove the limit and rely on the weight derived from the request.

(d) Why requests == limits is a good idea for memory and a bad one for CPU. Memory is not compressible: you cannot "go slower" in memory, either you have it or you are killed. Making request and limit equal guarantees that what the scheduler reserved for you is exactly what you can use, avoids OOM from node overcommit and gives the most stable service class. CPU, on the other hand, is compressible: it can be shared out over time. Making request and limit equal for CPU wastes the node's idle capacity — the pod cannot exploit peaks even when there are free cores — and causes avoidable throttling. The usual approach is a request matched to sustained use and a generous or even absent limit for trusted workloads.

Solution 3

(a) Volume and placement. Daily: 17,280,000 B ≈ 16.5 MiB. Annual: 17,280,000 × 365 ≈ 6.3 GB raw, and with 4:1 compression about 1.6 GB/year. Over five years, 7.9 GB compressed: a ridiculous amount, which reinforces that the criterion must not be space but latency and access pattern.

(b) IOPS.

• Reading by reading: 8,000 writes/s → 8,000 IOPS. At 3 IOPS/GB you would need a volume of 2,667 GB (about 2.7 TB) to store 16.5 MiB a day. It is absurd: you would be paying for an enormous volume just to buy IOPS.
• In batches of 512 readings (512 × 24 = 12,288 B per write): 8,000 / 512 ≈ 15.6 IOPS. At 3 IOPS/GB, ~6 GB is enough, and any reasonable volume (for example 100 GB, with 300 IOPS) is 20 times more than needed.

The conclusion is the lesson's own: batching writes is not a micro-optimization, it is a design requirement in the cloud. The cost of batching is a loss window — the readings in the current batch if the instance dies — which is bounded with a periodic fsync by time as well as by size, or by putting a persistent queue in front.

(c) Why the current day cannot go into objects, with three technical reasons:

1. There is no partial writing and no append. Adding a reading at the end would require rewriting the entire object (16.5 MiB) every time: 8,000 times a second is impossible and would cost a fortune in requests.
2. Latency is a thousand times worse. 20-100 ms per operation versus the microseconds of a local write; with 8,000 writes/s, nowhere near feasible.
3. There is no fsync, no locking and no POSIX semantics. You cannot guarantee the durability of a specific record as in 04-05, nor coordinate concurrent writers, nor use the file system's tools.

And a fourth, cost-based reason: object storage is billed per request, so 8,000 requests per second would be more expensive than everything else put together.

(d) 3-2-1 policy with immutability.

• 3 copies: the one on the block volume (production), the one in objects in the main region and the one in objects in another region.
• 2 different media: block volume and object storage, which are independent systems with different failure modes.
• 1 off-site: the copy in another region covers the loss of an entire region.
• Verification: a checksum on write and periodic checking; and above all, a quarterly restore test with the time measured, because an untested backup is not a backup (05-03).

What immutability adds. Ransomware with the instance's credentials can encrypt the volume and also delete or overwrite the objects, because it has the same permissions as the application. With versioning plus object lock in compliance mode, the provider rejects any deletion or modification until the retention expires, even if the request arrives with valid, administrator credentials. That puts the backup out of the attacker's reach, which is exactly the same reasoning that in 05-04 led to shipping the logs to an external collector with send-only credentials. The measure is completed with separate credentials for backups, different from the application's, and with alerts on any deletion attempt.

When the machine stops being a physical object and becomes an API call, what changes is not the administration but the mental model: the instance is provisioned in seconds, lives for hours or days, and can disappear without warning, with an individual availability commitment of around 99.5%. From that comes the rule that governs the whole lesson — do not try to keep the machine from failing; design so that its failure does not matter — and the cattle versus pets metaphor, with its uncomfortable consequences: no manual changes, no important data on the instance's disk, SSH as an exception rather than a tool, and reinstalling as the normal answer. With the honest caveat that not everything can be cattle: whatever has state is still a pet, and the goal is to concentrate it into as few pieces as possible.

The service models draw where the operating system's boundary lies: on IaaS it is entirely yours — and with it the patches, the hardening from 05-03, the firewall and the logs: a freshly created instance is as insecure as a freshly installed server — and as you climb to CaaS, PaaS and FaaS you gain operations and lose, very literally, this course's tools. Booting an instance chains together image, type, root volume, network and cloud-init, and the last of these is the piece that turns a generic template into your server: users with a public key, an explicit UID 990 because volumes are shared by number, mounts with noatime,nosuid,nodev declared rather than edited, and never a secret, because the user-data is readable from inside through the metadata service — the same endpoint that hands out the role's credentials and that turns any old SSRF into credential theft, as in the Capital One breach.

From that comes immutable infrastructure: instead of modifying servers, new images are built, which eliminates configuration drift by construction and turns rollback into something boring, in exchange for complicating debugging — hence the technique of removing from the load balancer without destroying, which connects with the evidence preservation from 05-04. Minimal, immutable operating systems take the idea to the extreme: with no shell there is no reverse shell, with no curl there is no second stage, with no compiler there is no local exploit; entire classes of attack are eliminated in exchange for having to debug with ephemeral containers that enter the process's namespaces, the nsenter from 06-02 industrialized.

In storage, the numbers rule: the network block volume is 5-10 times worse in latency than a local NVMe and has an IOPS ceiling you contract for, which turns write batching into a design requirement — from 8,000 IOPS down to 16 with batches of 512 readings; ephemeral storage is extremely fast and vanishes, so it is only good for what is rebuildable; and object storage is not a file system: no partial writing, no fsync, no locking, renaming that copies, and latencies of tens of milliseconds, which is why mounting it as a disk is almost always a mistake. Applied to Meteora: the current day on block, closed days in compressed objects transitioning to a cold class, and backups in another region with versioning and immutability, which is the only real defense against ransomware that has your credentials.

And the piece that closes the course's circle: orchestration is an operating system for the cluster. The pod is the process, the cluster scheduler is the scheduler from 02-02 one level up, the kubelet is the node's init, and a manifest's requests and limits translate literally into cpu.weight, cpu.max and memory.max in /sys/fs/cgroup — hence going over on CPU throttles you and going over on memory kills you with a 137. The scheduler places according to requests and not according to use, which makes inflated requests the number one cause of hugely expensive clusters at 15%. Around all that, Firecracker and gVisor show that the boundary between VM and container has stopped being binary (a microVM that boots in 125 ms by trimming everything QEMU emulated needlessly), autoscaling always trips over the cold start and oscillation, observability requires getting the logs off the instance because the instance may not exist when you go to read them, and cost becomes an architectural criterion in its own right.

We still have an entire family of operating systems that goes in exactly the opposite direction to all of this. Here we have talked about machines that are plentiful, that are created by API and thrown away when they get in the way; about scaling out when capacity runs short; about latencies measured in milliseconds and failures tolerated by retrying. But Meteora's weather stations — those embedded devices we decided back in 01-03 would run an RTOS — can do none of that: they have 64 KB of RAM, they run on a battery that has to last months, they cannot request another unit when they run short of memory, and if the sensor's sampling arrives 10 milliseconds late, the reading is wrong, not "slow to arrive". And at the other end of the same problem, the phone from which a user queries Meteora's API runs a Linux that has had to reinvent its IPC, its memory management and its permission model because the battery and the absence of swap change everything.

Two families with the same root we have studied and with constraints opposite to the server's. That is Mobile and Real-Time Operating Systems, and it closes the module.