Skip to content
LumenaryAcademy
Practice 85 min

Observability and autoscaling

Seeing inside a running cloud system, and letting it grow and shrink with demand — without waking anyone up for nothing.

Code tested · 5AI-draftedAwaiting review by a cloud engineer

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

Running blind, paying for idle

Picture a food-delivery service that runs in Lagos, Jakarta and Lima. Every evening, between about six and nine, orders arrive five times faster than at lunch and twenty times faster than at four in the morning. Two things can go wrong, and in the first year they usually both do.

The first is that the team cannot see. A customer posts that checkout has been "spinning" for ten minutes; every dashboard is green; nobody knows which of the twelve services behind checkout is slow, or whether it is slow for everyone or only for card payments in Peru. The second is that the team pays for capacity it does not use. To survive the dinner peak they run enough machines for the dinner peak all day long, and most of that fleet idles for twenty hours out of twenty-four.

This lesson is about the two practices that fix those problems, and about how they depend on each other. Observability is the ability to understand what a running system is doing from what it emits. Autoscaling changes how much capacity a system has, automatically, in response to what it measures — so it is only as good as the measurements behind it. Both put Lumenary CS's reliability engineering — SLOs, error budgets, alerting on symptoms, blameless incident response — into daily practice on a cloud. We will build a small tracer, simulate autoscaling policies against a day of traffic, and test alerting rules against a month of synthetic incidents.

DefinitionsH

Metrics, logs and traces

Definition— Observability

The degree to which you can explain a system's internal behaviour from its external outputs — including behaviour nobody anticipated — without changing and redeploying it. The word comes from control theory, where a system is observable if its internal state can be reconstructed from its outputs.

Monitoring asks known questions ("is the error rate above 1%?"); observability lets you answer new ones at three in the morning ("is it only Android users in Peru paying by card?"). It rests on three kinds of telemetry, called signals.

  • Metrics are numbers aggregated over time: a counter that only goes up (requests served, bytes sent), a gauge that goes up and down (queue length, memory in use), and a histogram that counts observations in buckets (how many requests took under 50 ms, under 100 ms, …), from which percentiles are estimated. Each series carries labels (in OpenTelemetry, attributes) such as service, endpoint and status. Metrics are cheap to store and fast to query, so they drive dashboards, alerts and autoscalers.
  • Logs are timestamped records of individual events. Structured logs — one JSON object per event, with fields rather than prose — can be searched and aggregated like data. They are rich, and they are the most expensive signal at scale: a busy service can write terabytes a day, and most providers bill per gigabyte ingested and per month retained.
  • Traces follow one request across every service it touches. A trace is a tree of spans; each span records one operation (an HTTP call, a database query) with its start, duration, attributes and the identifier of its parent span. Traces answer the question metrics cannot: where did this request's time go?

Tracing works only if every service passes the trace's identity on to the next. This is context propagation: when a service makes an outgoing call, it injects the trace identifier and its own span identifier into the request — for HTTP, the traceparent header standardised by the W3C Trace Context recommendation — and the receiving service extracts them and makes its span a child of the caller's. The design descends from Google's Dapper system, described in a 2010 report.

OpenTelemetry (OTel), a Cloud Native Computing Foundation project formed in 2019 by merging the OpenTracing and OpenCensus projects, is the vendor-neutral standard for producing all three signals: APIs and SDKs for most languages, automatic instrumentation for common frameworks, semantic conventions (agreed attribute names such as http.response.status_code, so that dashboards work across services), a wire protocol (OTLP) and the Collector, a pipeline that receives, filters, samples and forwards telemetry to whichever backends you choose. Instrument once, and the back end can change later without touching application code.

The tracer below is a toy version of what an OpenTelemetry SDK does: each simulated service extracts the traceparent it receives, records a span, and injects a new header into its own calls. Then we read the trace.

python
import random

rng = random.Random(2026)
def new_id(bits):                  # W3C Trace Context: 128-bit trace ids, 64-bit span ids
    return f"{rng.getrandbits(bits):0{bits // 4}x}"

exported = []                      # what an SDK would send to a tracing backend

class Span:
    def __init__(self, name, service, traceparent, start_ms):
        if traceparent:            # extract: continue the caller's trace
            version, trace_id, parent_id, flags = traceparent.split("-")
        else:                      # no header: this request starts a new trace
            trace_id, parent_id = new_id(128), None
        self.trace_id, self.parent_id, self.span_id = trace_id, parent_id, new_id(64)
        self.name, self.service, self.start, self.end = name, service, start_ms, None
        exported.append(self)

    def traceparent(self):         # inject: the header put on every outgoing call
        return f"00-{self.trace_id}-{self.span_id}-01"

# Five services, each a function (incoming header, arrival time in ms) -> finish time.
def stock_db(tp, t):
    s = Span("SELECT stock", "orders-db", tp, t); s.end = t + 18; return s.end

def fraud_check(tp, t):
    s = Span("POST /score", "fraud", tp, t); s.end = t + 35; return s.end

def card_network(tp, t):
    s = Span("POST /authorise", "card-network", tp, t); s.end = t + 260; return s.end

def payments(tp, t):
    s = Span("POST /payments", "payments", tp, t)
    t = fraud_check(s.traceparent(), t + 2)
    t = card_network(s.traceparent(), t + 1)
    s.end = t + 4
    return s.end

def orders(tp, t):
    s = Span("POST /orders", "orders", tp, t)
    t = stock_db(s.traceparent(), t + 1)
    t = payments(s.traceparent(), t + 3)
    s.end = t + 6
    return s.end

def cart(tp, t):
    s = Span("GET /cart", "cart", tp, t); s.end = t + 31; return s.end

def gateway(t):
    s = Span("POST /checkout", "gateway", None, t)
    t = cart(s.traceparent(), t + 2)
    t = orders(s.traceparent(), t + 2)
    s.end = t + 3
    return s

root = gateway(0)
print("header sent from the gateway to the cart service:")
print("  traceparent:", root.traceparent(), "\n")

by_id = {s.span_id: s for s in exported}
def depth(s):
    return 0 if s.parent_id is None else 1 + depth(by_id[s.parent_id])
def self_time(s):                  # time not covered by the span's children (calls are sequential here)
    return (s.end - s.start) - sum(c.end - c.start for c in exported if c.parent_id == s.span_id)

total, width = root.end - root.start, 40
print(f"trace {root.trace_id[:8]}...  {total} ms in total")
print(f"{'span':24s} {'service':13s} {'start':>5s} {'ms':>4s} {'self':>4s}  timeline")
for s in sorted(exported, key=lambda s: s.start):
    a, b = round(s.start / total * width), max(round(s.end / total * width), round(s.start / total * width) + 1)
    bar = " " * a + "#" * (b - a) + " " * (width - b)
    name = "  " * depth(s) + s.name
    print(f"{name:24s} {s.service:13s} {s.start:5d} {s.end - s.start:4d} {self_time(s):4d}  |{bar}|")

worst = max(exported, key=self_time)
print(f"\nmost self time: {worst.name} in {worst.service}, "
      f"{self_time(worst)} of {total} ms ({self_time(worst) / total:.0%})")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The trace answers in one glance what twelve dashboards could not: checkout is slow because the card network's authorisation takes 260 of its 368 ms. Self time — a span's duration minus its children's — points at the work done in a span rather than merely beneath it. (Here every call is sequential; with parallel calls, the critical path is what matters.) Real systems produce far too many traces to keep them all, so they sample: head sampling decides at the start of a request (keep 1 in 100), while tail sampling, usually done in the Collector, waits until a trace is complete and keeps all the slow and failed ones.

ExplanationH

SLOs from cloud telemetry

Lumenary CS's reliability engineering lesson defined the tools; here is how they meet the telemetry above. A service level indicator is a ratio of good events to valid events, and on a cloud platform the most faithful place to count them is usually the load balancer or API gateway, which sees every request including those that never reached a healthy server. An availability SLI counts responses that are not server errors; a latency SLI counts requests faster than a threshold, read from a histogram. An SLO sets a target over a window (99.9% over 30 days), and the error budget — 0.1% of requests, about 43 minutes of total outage — is what the team may spend on deploys, experiments and bad luck.

Two cloud-specific points matter. Your service depends on the provider's services, and availabilities multiply: three dependencies in series, each at 99.95%, give at best 99.85%. A provider's SLA promises a partial refund when a service misses a stated level, not that it will not; surviving a failed zone or managed service is your design's job. And SLOs connect to autoscaling and cost: they say how much headroom you must buy, and headroom has a price.

IntuitionH

Autoscaling as a control loop

Horizontal scaling changes the number of instances (virtual machines, containers or functions); vertical scaling changes the size of each. Horizontal scaling is the cloud's native move, because stateless servers behind a load balancer are interchangeable; vertical scaling usually needs a restart, and stateful systems such as databases scale in their own, slower ways (read replicas, sharding, storage that grows automatically).

An autoscaler is a feedback controller. Every evaluation period it reads a metric, compares it with a goal, and asks the platform for more or fewer instances. The platform launches them — which takes time — and the new capacity eventually changes the metric.

RequestsInstancesMetricsScaling policyloadCPU, req/s,queue lengthlate by1–5 minlaunch or stop instances: new ones serve only after booting (minutes)
An autoscaler is a feedback loop with two delays: the metric reaches the policy late, and new instances serve traffic only after they boot.

Three families of policy fill in the box on the right.

  • Target tracking keeps a metric near a set point, the way a thermostat keeps a room near 21 °C. If 12 instances run at 84% CPU and the target is 60%, the group needs about 12×84/60=16.812 \times 84 / 60 = 16.8, so 17 instances. Kubernetes' Horizontal Pod Autoscaler uses exactly this proportional rule, with a tolerance (10% by default) inside which it does nothing.
  • Step scaling maps bands of the metric to fixed changes: above 80% add 30%, between 70% and 80% add 10%, below 40% remove 10%, then wait out a cooldown before acting again.
  • Predictive scaling forecasts demand from history — most services repeat a daily and weekly rhythm — and launches capacity before it is needed, by at least the boot time. It is combined with a reactive policy, which still handles whatever the forecast missed. Scheduled scaling is its manual cousin: "at 17:30 on match days, at least 40 instances".

Scaling in needs more caution than scaling out. Removing capacity too early means scaling out again minutes later (flapping), and an instance must first be drained — no new requests, in-flight ones allowed to finish. So autoscalers scale out quickly and in slowly; Kubernetes by default waits until its scale-down recommendation has been stable for five minutes.

ComputationC

Policies against a day of traffic

The simulation below gives the delivery service a week of history and one more day to scale. Each instance serves 100 requests per second at 100% CPU and takes 8 minutes from launch to serving traffic; the metric reaches the policy a minute late, and CPU cannot read above 100%. Requests beyond capacity fail, and minutes above 85% utilisation count as "hot", because queueing makes latency climb steeply there. At 18:30 marketing sends a push notification that nobody told the operations team about.

python
import math
import random

rng = random.Random(42)
DAY = 1440                                     # minutes

def bump(minute, centre_h, width_h, height):
    d = (minute / 60 - centre_h) / width_h
    return height * math.exp(-d * d)

def traffic(days, promo_at=None):
    """Requests per second, minute by minute: a quiet night, a lunch peak, a bigger dinner peak."""
    out = []
    for day in range(days):
        level = 1 + 0.04 * rng.uniform(-1, 1)                    # some days are a little busier
        for m in range(DAY):
            rps = 250 + bump(m, 12.5, 1.6, 1900) + bump(m, 19.5, 1.9, 3300)
            out.append(rps * level * (1 + 0.03 * rng.gauss(0, 1)))
    if promo_at is not None:                                     # a push notification ops did not expect
        for k in range(45):
            out[promo_at + k] *= 1 + 0.8 * min(1, k / 2) * math.exp(-k / 30)
    return out

history = traffic(7)                                             # the previous seven days
PROMO = 18 * 60 + 30
today = traffic(1, promo_at=PROMO)

PER_INSTANCE = 100       # requests per second one instance serves at 100% CPU
BOOT = 8                 # minutes from launch to serving traffic
LO, HI = 2, 80           # smallest and largest allowed group

def run(policy):
    serving, pending, state = 10, [], {}                         # pending: minutes when launches finish
    billed = failed = total = hot = hot_outside = 0
    util_seen, sizes = today[0] / (serving * PER_INSTANCE), []
    for m, rps in enumerate(today):
        serving += pending.count(m)
        pending = [t for t in pending if t > m]
        current = serving + len(pending)
        desired = max(LO, min(HI, policy(m, util_seen, serving, current, state)))
        if desired > current:
            pending += [m + BOOT] * (desired - current)          # new instances must boot first
        elif desired < current:
            cancel = min(current - desired, len(pending))        # cancel launches first,
            pending = pending[:len(pending) - cancel]
            serving -= min(current - desired - cancel, serving - LO)   # then stop serving ones
        util = rps / (serving * PER_INSTANCE)
        failed += max(0.0, rps - serving * PER_INSTANCE) * 60    # requests beyond capacity fail
        total += rps * 60
        hot += util > 0.85                                       # latency suffers above about 85%
        hot_outside += util > 0.85 and not PROMO <= m < PROMO + 45
        billed += serving + len(pending)                         # booting instances are billed too
        util_seen = min(util, 1.0)                               # CPU reads at most 100%, a minute late
        sizes.append(serving)
    return billed / 60, failed / total, hot, hot_outside, sizes

def fixed(n):
    return lambda m, u, serving, current, st: n

def target_tracking(target, window=10):
    def policy(m, u, serving, current, st):
        want = math.ceil(serving * u / target)                   # what the measured load needs
        recent = st.setdefault("recent", [])
        recent.append(want)
        del recent[:-window]
        return want if want >= current else max(recent)          # scale in only after 10 calm minutes
    return policy

def step_scaling(m, u, serving, current, st):
    if m < st.get("cooldown_until", 0):
        return current
    if u > 0.80:   change = max(2, math.ceil(0.3 * current))
    elif u > 0.70: change = max(1, math.ceil(0.1 * current))
    elif u < 0.40: change = -max(1, math.floor(0.1 * current))
    else:          return current
    st["cooldown_until"] = m + 3
    return current + change

forecast = [sum(history[d * DAY + m] for d in range(7)) / 7 for m in range(DAY)]
def predictive(target):
    reactive = target_tracking(target)
    def policy(m, u, serving, current, st):
        ahead = forecast[min(m + BOOT, DAY - 1)]                 # plan for when launches will be ready
        return max(math.ceil(ahead / (PER_INSTANCE * target)), reactive(m, u, serving, current, st))
    return policy

policies = {
    "fixed for last week's peak": fixed(math.ceil(max(history) / (PER_INSTANCE * 0.6))),
    "fixed for the average":      fixed(math.ceil(sum(history) / len(history) / PER_INSTANCE)),
    "step scaling":               step_scaling,
    "target tracking, 60% CPU":   target_tracking(0.60),
    "target tracking, 80% CPU":   target_tracking(0.80),
    "predictive + target, 80%":   predictive(0.80),
}
PRICE = 0.10   # USD per instance-hour: a made-up round number, for comparison only
print(f"today's peak {max(today):,.0f} req/s (promotion at 18:30); last week's peak {max(history):,.0f}\n")
print(f"{'policy':27s} {'inst-h':>6s} {'USD':>6s} {'failed':>7s} {'min>85%':>8s} {'not promo':>9s}")
sizes = {}
for name, policy in policies.items():
    hours, fail, hot, outside, sizes[name] = run(policy)
    print(f"{name:27s} {hours:6.0f} {hours * PRICE:6.2f} {fail:7.2%} {hot:8d} {outside:9d}")

print("\nthe promotion: requests per second and instances serving")
print("time   req/s  target 60%  predictive 80%")
for m in range(PROMO - 4, PROMO + 16, 2):
    print(f"{m // 60}:{m % 60:02d}  {today[m]:6.0f}  {sizes['target tracking, 60% CPU'][m]:10d}"
          f"  {sizes['predictive + target, 80%'][m]:14d}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
02,0004,0006,0008,00000:0006:0012:0018:0024:00time of day (requests per second on the vertical axis)demandcapacity of instances serving
The simulated day under target tracking at 60% CPU: demand (gold) and the full capacity of the instances serving (blue). Capacity follows the lunch and dinner peaks with a margin, but the 18:30 promotion arrives faster than new instances can boot.

A fleet sized for last week's peak never fails but costs USD 158 a day at this made-up price; one sized for the average costs USD 24 and fails 41% of requests. Every autoscaling policy costs between USD 32 and 46, much nearer the second than the first, with nearly the first's reliability: target tracking at 60% cuts the bill by 73%. None survives the promotion untouched: demand nearly doubles within two minutes, and nothing launched after 18:30 can serve before 18:38.

The last two rows show what forecasting buys. Raising the target to 80% saves another quarter of the cost, but with only 20% headroom the group cannot keep up with the ordinary evening ramp, and 83 minutes run hot outside the promotion. Adding the forecast floor at the same 80% target removes all of those hot minutes, so predictive scaling plus target tracking costs 22% less than target tracking at 60% with the same behaviour on an ordinary day. What it cannot do is predict the push notification; with less headroom, it fails more requests during the surprise than the 60% policy does. Headroom is insurance against what you did not predict, forecasts pay for what you can, and the cheapest fix for the promotion is not technical at all: tell the operations team, and schedule capacity for 18:25.

ExplanationH

Autoscaling on real platforms

Every major platform offers the same ideas under different names, and each documents them in full. The table is a map, as of 2026, not a ranking.

PlatformUnit that scalesPolicy types offered
Amazon Web Services: EC2 Auto ScalingAuto Scaling groups of VMstarget tracking, step and simple scaling, scheduled actions, predictive scaling
Microsoft Azure: autoscaleVirtual Machine Scale Sets, App Service and othersmetric threshold rules, schedules and profiles, predictive autoscale (CPU, scale-out only)
Google Cloud: autoscalermanaged instance groupstarget utilisation (CPU, load-balancer capacity, monitoring metrics), schedules, predictive autoscaling
Alibaba Cloud: Auto Scalingscaling groups of ECS instances or elastic container instancessimple, step, target tracking and predictive scaling rules, scheduled and event-triggered tasks
Kubernetes (open source, on any provider)pods, and the nodes under themHorizontal Pod Autoscaler (proportional), Vertical Pod Autoscaler, node autoscalers such as Cluster Autoscaler and Karpenter, KEDA for event-driven scaling

On Kubernetes, which the Kubernetes lesson introduced, a target-tracking policy is a short manifest. This one (not run here) keeps the checkout Deployment near 60% CPU, adds up to 100% more pods per minute, and removes at most 10% per minute, only after the recommendation has been lower for ten minutes:

yaml
# Not run here: a Kubernetes manifest (autoscaling/v2), applied with kubectl apply -f.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  minReplicas: 4
  maxReplicas: 80
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60     # percent of each pod's CPU request
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

Pods can only be scheduled onto nodes that exist, so a pod autoscaler on its own simply creates pending pods when the cluster is full; a node autoscaler adds machines underneath, and its boot delay adds to the pods' own. Functions and other serverless platforms, the subject of Serverless and event-driven design, scale per request and down to zero, trading the boot delay for a cold start on the first request after a quiet spell.

Experience adds a few rules that no policy type removes:

  • Shorten the loop. Smaller images, pre-baked machine images and warm pools of stopped, pre-initialised instances cut the boot delay that every policy fights.
  • Set the maximum deliberately. It is a cost guardrail and protects whatever sits behind the fleet: a hundred new web servers can open enough connections to topple a database. Provider quotas per region also cap you, and raising them takes a request in advance.
  • Scale on what predicts trouble: requests per instance for web tiers, backlog per worker for queue consumers, never on a metric the fleet's own size does not change.
  • Test scaling: load tests that ramp as fast as real traffic, and drills for when a zone or instance type has no capacity left.
ComputationC

Designing alerts that deserve to wake someone

An alert that pages a person should be urgent, actionable and about something users feel; everything else becomes a ticket for working hours or a line on a dashboard. Error budgets make "something users feel" measurable: a page should mean that the budget is burning fast enough to matter. The burn rate is the observed error ratio divided by the ratio the SLO allows, so a burn rate of 1 spends the budget exactly over the window. The SRE workbook derives a scheme that pages when the burn rate exceeds 14.4 over the last hour (2% of a 30-day budget in one hour) or 6 over the last six hours, and opens a ticket when it exceeds 1 over three days. Each long window is paired with a short one (five and thirty minutes), so the alert also stops soon after the problem does.

The program below runs three alerting rules over a synthetic month with four incidents and a dozen ordinary one- or two-minute hiccups, and counts how often each would notify someone.

python
import numpy as np

rng = np.random.default_rng(5)
DAYS, M = 30, 1440
minute = np.arange(DAYS * M)
hour = (minute % M) / 60
requests = (2_000 + 18_000 * np.exp(-((hour - 14) / 4.5) ** 2)).round()   # per minute: quiet nights
bad_ratio = np.full(minute.size, 0.0002)                                  # 0.02% background errors

incidents = [   # (name, day, start hour, minutes, error ratio)
    ("3-minute blip at night",   3,  2.2,    3, 0.04),
    ("bad deploy",               9, 14.0,  240, 0.02),
    ("major outage",            17, 19.5,   25, 0.30),
    ("slow leak over two days", 23,  0.0, 2880, 0.0025),
]
hiccups = np.zeros(minute.size, bool)
for start in rng.choice(DAYS * M - 5, size=12, replace=False):   # everyday hiccups: a pod restarts,
    length = rng.integers(1, 3)                                    # a cache is cold for a minute
    bad_ratio[start:start + length] = rng.uniform(0.015, 0.06)
    hiccups[start:start + length] = True
spans = []
for name, day, h, length, ratio in incidents:
    start = day * M + int(h * 60)
    bad_ratio[start:start + length] = ratio
    spans.append((name, start, start + length))
errors = rng.binomial(requests.astype(int), bad_ratio)

SLO = 0.999
def burn(window):
    """Burn rate over the trailing `window` minutes, ending at each minute."""
    e, r = np.cumsum(errors), np.cumsum(requests)
    e_w = e - np.concatenate([np.zeros(window), e[:-window]])
    r_w = r - np.concatenate([np.zeros(window), r[:-window]])
    return (e_w / r_w) / (1 - SLO)

def notifications(firing, quiet=60):
    """Minutes at which someone is notified: firing, after `quiet` minutes without firing."""
    out, last = [], -10**9
    for t in np.flatnonzero(firing):
        if t - last > quiet:
            out.append(int(t))
        last = t
    return out

H = 60
alerts = {
    "page: 1-min errors > 1%":  {"page": burn(1) > 10},
    "page: 5-min errors > SLO": {"page": burn(5) > 1},
    "multiwindow burn rate":    {"page": ((burn(H) > 14.4) & (burn(5) > 14.4)) |
                                         ((burn(6 * H) > 6) & (burn(30) > 6)),
                                 "ticket": (burn(3 * 24 * H) > 1) & (burn(6 * H) > 1)},
}

budget = (1 - SLO) * requests.sum()
print(f"30-day error budget: {budget:,.0f} failed requests\n")
for name, a, b in spans:
    print(f"{name:25s} used {errors[a:b].sum() / budget:6.1%} of the budget")

print(f"{'12 hiccups of 1-2 minutes':25s} used {errors[hiccups].sum() / budget:6.1%} of the budget, together")
for rule, kinds in alerts.items():
    sent = {kind: notifications(firing) for kind, firing in kinds.items()}
    print(f"\n{rule}: {len(sent['page'])} pages, {len(sent.get('ticket', []))} tickets")
    for name, a, b in spans:
        hits = sorted((t, kind) for kind in sent for t in sent[kind] if a <= t < b + 60)
        if hits:
            t, kind = hits[0]
            print(f"   {name:25s} {kind:6s} after {t - a:4d} min, {errors[a:t + 1].sum() / budget:5.1%} of budget spent")
        else:
            print(f"   {name:25s} not noticed")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The two threshold rules notice almost everything within the first minute — including twelve hiccups and a three-minute blip at night that together cost under 2% of the budget, and that were over before anyone could have opened a laptop. Thirteen or fourteen pages a month, most of them needing no action, is how a rotation learns to ignore its pager. The 1% rule also never notices the slow leak that quietly spends a sixth of the budget over two days, while the SLO-threshold rule pages for it at one minute past midnight, waking someone for a problem that could have waited for the morning.

The burn-rate rules page twice: for the major outage after 3 minutes, and for the bad deploy after 43 minutes, by which time it has spent about 5% of the budget. They stay silent for the blip and the hiccups, and the slow leak becomes a ticket after about 29 hours, having spent under 9% of the month's budget. That is the trade the scheme makes deliberately — slower detection of moderate problems in exchange for pages that are almost always worth answering — and the thresholds can be tuned when the trade is wrong for a service. Two more habits help: every alert links to a runbook that says what to check first, and alerts are grouped, so that one broken dependency produces one page rather than forty.

ExplanationH

Incident response on a cloud platform

The structure of incident response — an incident commander who coordinates rather than debugs, an operations lead who makes changes, a communications lead, one shared record, mitigation before diagnosis, a blameless postmortem — is in the reliability engineering lesson and the SRE book's chapter on managing incidents. Here is how the simulated promotion might have run, with the telemetry above doing its work.

Example— The promotion, as an incident

18:30 The push notification goes out. 18:31 Error ratio jumps; load-balancer metrics show the web tier rejecting requests; CPU across the fleet reads 100%. 18:33 The fast burn-rate alert pages the on-call engineer, Thandiwe, with a link to the checkout runbook. 18:35 She declares an incident and takes the commander role; Rafael joins as operations lead. The trace view shows requests failing at the gateway, not deep inside payments, which rules out the card network. 18:36 Mitigation: Rafael raises the group's minimum size so that capacity no longer depends on the policy catching up, and turns on load shedding for the recommendations widget to protect checkout. 18:38–18:44 New instances come into service; errors fall back inside the SLO. 18:50 The communications lead posts an all-clear on the status page. Next week The postmortem's action items: a calendar of promotions shared with the operations team, scheduled scaling ten minutes before each, a warm pool to halve the boot delay, and scaling on requests per instance instead of CPU.

Cloud incidents add one early question: is it us or them? Each provider publishes a public status page and a personalised health dashboard for the resources in your account — the AWS Health Dashboard, Azure status and Azure Service Health, the Google Cloud Service Health dashboard, and Alibaba Cloud's status page. Public pages often lag the start of a problem, so your own telemetry, broken down by region and zone, is usually the first and best evidence. If a provider's zone or region is failing, the mitigations are the ones your architecture prepared — shift traffic, fail over the database, degrade gracefully — and a failover never practised should not be expected to work. The telemetry pipeline must also survive the incident it is meant to explain: if your dashboards run in the same region as the system that is down, you may be blind exactly when you need to see.

Further reading

The professional view

As of 2026, most organisations instrument with OpenTelemetry and choose where to send the data separately. Open-source back ends include Prometheus for metrics, Jaeger for traces, the Grafana projects (Grafana for dashboards, Loki for logs, Tempo for traces, Mimir for metrics) and OpenSearch for logs and search. Each large provider runs a managed observability suite that also accepts OpenTelemetry data: Amazon CloudWatch with AWS X-Ray, Azure Monitor with Application Insights, Google Cloud Observability (Cloud Monitoring, Logging and Trace), and on Alibaba Cloud CloudMonitor, Simple Log Service, Application Real-Time Monitoring Service and Managed Service for OpenTelemetry. Commercial observability vendors compete in the same space. Self-hosting gives control and can cost less at scale, at the price of running another critical system; managed suites remove that work and bill by volume ingested, stored and queried.

Telemetry can grow to a noticeable fraction of the cost of the system it observes, so teams sample traces, drop debug logs at the Collector, and keep high-resolution metrics for days and downsampled ones for months. Active areas in 2026 include profiles as a fourth signal (continuous, low-overhead sampling of where CPU time and memory go), kernel-level instrumentation with eBPF, and machine-learning help with incident triage — useful for correlating signals, but no substitute for a well-defined SLO. The cost side returns in Cost and FinOps.

Common misconceptionsH

Common misconceptions

Misconception

"Observability means having lots of dashboards." Dashboards answer the questions someone thought of in advance. Observability is the ability to ask new ones — which needs high-quality traces and structured events with the right attributes, not more graphs.

Misconception

"With autoscaling we no longer need capacity planning." Autoscaling moves capacity planning from "how many machines?" to "how fast can we add them, up to what maximum, within which quotas, and what breaks behind the fleet when we do?" Boot time, quotas and dependencies are still planned.

Misconception

"Autoscaling protects against sudden spikes." It protects against spikes slower than its loop. A spike that arrives in two minutes, against an eight-minute boot, is absorbed by headroom, load shedding or scheduled capacity — or not at all.

Misconception

"Log everything; storage is cheap." Ingestion, indexing and retention are billed per gigabyte and add up quickly, and logs full of noise are slower to search during an incident. Logs should also never contain secrets or more personal data than needed, as Cloud security and identity discusses.

Exercises

Exercises

1. Observability and autoscaling

Warm-upHuman

A customer in Lima reports that one checkout took 9 seconds at 19:42. Your dashboards show a 99th-percentile checkout latency of 800 ms for that minute and no rise in errors. Which signal is most likely to show where that request spent its 9 seconds?

Choose one

2. Observability and autoscaling

CoreHuman

A latency histogram has 12 buckets (including the last, unbounded one), and each label combination also stores a sum and a count series. It is labelled by endpoint (25 values), status_class (5 values) and region (3 values). How many time series does it create?

3. Observability and autoscaling

CoreHuman

On an ordinary evening, demand for a service rises by at most 30 requests per second every minute. New instances take 8 minutes from launch to serving traffic, and the autoscaler's metric arrives 1 minute late. At least how many requests per second of spare capacity (headroom) must the group carry so that demand never outruns capacity while new instances are on their way?

4. Observability and autoscaling

CoreComputational

Write desired_replicas(current, metric, target, min_replicas, max_replicas, tolerance=0.1), the proportional rule used by target-tracking autoscalers such as Kubernetes' Horizontal Pod Autoscaler:

  • If metric / target is within tolerance of 1 (that is, ∣metric/target−1∣≤|\text{metric}/\text{target} - 1| \le tolerance), keep current.
  • Otherwise ask for ⌈current×metric/target⌉\lceil \text{current} \times \text{metric} / \text{target} \rceil replicas.
  • Never return fewer than min_replicas or more than max_replicas.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

5. Observability and autoscaling

StretchComputational

errors and requests are lists of per-minute counts, oldest first, with at least 60 minutes of data. The starter pages whenever the burn rate over the last 5 minutes exceeds the threshold, so it wakes someone for every short blip.

Fix should_page(errors, requests, slo=0.999, threshold=14.4) so that it returns True only if the burn rate exceeds threshold over both the last 60 minutes and the last 5 minutes. The burn rate over a window is (errors in the window ÷ requests in the window) ÷ (1 − slo).

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

6. Observability and autoscaling

CoreHuman

A checkout service has a 99.9% availability SLO over 30 days. Which of these situations should page the on-call engineer (rather than open a ticket or simply appear on a dashboard)? Select all that apply.

Choose all that apply

7. Observability and autoscaling

CoreHuman

For which of these workloads would adding predictive scaling to a target-tracking policy help most?

Choose one

Further reading and resources