Cloud autoscaling: buy capacity only for a real surge

An autoscaler adds instances when load per instance is above target and removes them when it is below. Scaling on the latest sample buys capacity for one-minute blips, thrashes on oscillating load, and acts on half a picture after a metrics outage. This page shows an autoscaler that scales out when the last fifteen minutes show a coherent surge across zones, and leaves capacity alone otherwise. It is written for SRE and platform owners; the runnable example id is cloud_autoscaling.

At a glance

Decision	Scale out, scale in, or leave capacity unchanged this tick.
Owner	The autoscaler; the platform team that carries the pager and the bill.
Today's rule	Scale whenever the latest sample crosses the target (with a cooldown).
What this adds	A check that the recent window shows a consistent pattern across zones and minutes, and that it looks like past scale-outs that were needed.
Demo outcome	5 ticks: 2 acted (one scale-out, one scale-in), 3 held.
Example id	cloud_autoscaling (uv run grounded-decision examples cloud_autoscaling)

The decision today

The autoscaler compares the latest requests-per-instance sample with a target (80 in the demo) and scales out above it, in below it. Cooldown timers and "N of M datapoints" rules patch the flapping. They are fixed counts and know nothing about whether this load pattern looks like surges that actually needed capacity.

Mistake	What it costs
Scale out on a blip	Instances billed for their minimum period, then scaled in again.
Scale on oscillating load	A fleet that thrashes, and a retry storm hidden instead of fixed.
Hold during a real surge	Latency and errors until the surge is confirmed or a person scales.

Where today's rule goes wrong

Four busy signals arrive in the demo and the naive rule scales out on all four. Only the first should have bought capacity:

• Sustained surge. Every zone has been at 130 requests per instance for the last ten minutes. Real.
• One-minute blip. One sample of 140 in zone_a at the latest minute; the other 44 samples sit at a baseline of 45.
• Oscillating load. Every zone flips between 110 and 50 each minute (a retry-storm pattern).
• Metrics gap. Load is 130 in every zone, but the metrics pipeline only recovered five minutes ago, so the window has 15 samples instead of 45. All 15 are above target; there is not yet enough history to tell a surge from a spike.

A fifth signal, quiet after fifteen minutes at 45 in every zone, is the normal scale-in.

What the system checks, in your terms

The evidence is a load sample graph over the last fifteen minutes: one sample per availability zone per minute, linked to the same minute in the other zones and to the next minute in the same zone, and each marked above or below target.

Check	Question in platform terms	Data it reads	Fails when	What a hold means
Neighbour agreement	Is the above/below pattern consistent across zones and consecutive minutes?	Above/below target per sample, links	Load flips minute to minute	The load is oscillating; adding capacity will not settle it.
Knife-edge	Is there enough history to read the pattern?	Number of samples in the window	The window is short (here: under about 12 of the 15 minutes)	Not enough data yet; decide again next tick.
Unusual cases	Does the share of samples contradicting the signal look like past scale-outs that were needed?	Share of below-target samples on busy; 30 justified scale-outs	Far more contradicting samples than any justified scale-out had	The signal is one sample against a calm window.
Policy shape and progress	Is the scale-out/scale-in policy well formed?	The policy, once at build	Never here	— (build-time checks; the progress check is weak by construction, see progress).

What these checks do not verify

• Neighbour agreement is direction-blind. On the blip, the 44 baseline samples agree with each other, so that check passes. It asks whether samples predict their neighbours, not whether they support scaling out.
• Direction comes from the unusual-cases check, because the candidate score is the share of samples contradicting the event (below-target samples on busy, above-target on quiet). That holds the blip, and it would hold a quiet signal sent during the sustained surge. If the calibration set is empty, that check passes by default and this protection disappears; set Limits(require_conformal_calibration=True) in production.
• The knife-edge check is really an "enough history" guard here. On a full window the link density is fixed by the grid (0.088 on every row), so the check cannot react to load. It reacts only to window length: in this demo, windows of 10 minutes or less fail it and 12 minutes or more pass. If that is all you need, a plain "require at least N minutes of samples" rule says the same thing more clearly.

Data you provide

Input	Source	Refresh	Minimum / quality bar
(zone, minute, requests_per_instance) rows for the last 15 minutes	CloudWatch / Prometheus range query	Every evaluation tick	One sample per zone per minute; gaps shorten the window and cause holds.
Event (busy / quiet)	Your existing metric rule on the latest sample	Every tick	—
Justified scale-out shares	Autoscaler action log joined with utilisation: scale-outs whose instances stayed above 50% for their first billing hour	Weekly	At least 9 at α = 0.10; the demo uses 30 (0.00 to 0.58). Judged an hour later; never include the current tick.
Justified scale-in shares	Scale-ins not reversed within an hour	Weekly	At least 9; the demo uses 30 (0.000 to 0.116).

Worked example

Tick	Window	What the checks found	Verdict	What the autoscaler does
Sustained surge	30 of 45 samples above target, last 10 minutes in every zone	Consistent across zones and minutes; contradicting share (0.33) like past justified scale-outs	Act scale_out	Adds instances.
One-minute blip	1 of 45 above target	Consistent, but 44 of 45 samples contradict the signal — more than any justified scale-out	Hold, proposed scale_out	Leaves capacity unchanged.
Oscillating load	24 of 45 above, flipping each minute	Consecutive minutes disagree: no consistent pattern	Hold, proposed scale_out	Leaves capacity unchanged; flag a possible retry storm.
Metrics gap	15 of 15 above, only 5 minutes of data	Not enough history	Hold, proposed scale_out	Leaves capacity unchanged; re-evaluates next tick.
Quiet	0 of 45 above	Consistent; nothing contradicts	Act scale_in	Removes instances.

"Hold" is the record verdict abstain: the proposal stays on the record and is not carried out.

The numbers (from uv run grounded-decision examples cloud_autoscaling)

Cut-off \(p^{*} = 0.25\) on every row.

Tick	\(n\)	Links	\(d\)	\(\lambda\)	Strength \(d\lambda^2\) (> 1.05)	\(\hat p\)	Knife-edge	Contradicting share	Unusualness \(p\) (> 0.10)
Surge	45	87	3.87	0.931	3.35 pass	0.088	outside, pass	0.333	0.452 pass
Blip	45	87	3.87	0.931	3.35 pass	0.088	outside, pass	0.978	0.032 fail
Oscillating	45	87	3.87	0.034	0.005 fail	0.088	outside, pass	0.467	0.226 pass
Metrics gap	15	27	3.6	1.0	3.6 pass	0.257	inside, fail	0.0	1.0 pass
Quiet	45	87	3.87	1.0	3.87 pass	0.088	outside, pass	0.0	1.0 pass

\(d\) is average links per sample, \(\lambda\) how much more often linked samples share a label than chance, \(\hat p\) the link density. At \(n = 45\) the knife-edge half-width is 0.111 and \(|0.088 - 0.25| = 0.162\) is outside; at \(n = 15\) it is 0.219 and \(|0.257 - 0.25| = 0.007\) is inside.

--json summary: naive_acts: 5, grounded_acts: 2, grounded_abstains: 3, blocked_overconfident_acts: 3.

What happens next

• Act. Call the scaling API with record.action.
• Hold. Leave capacity unchanged, emit record.reason as a metric (autoscaler.hold tagged by failed check), and decide again on the next tick. A real surge keeps producing busy signals and is acted on once the window shows it consistently.
• Bound the hold. A hold must not last through a real outage. Add a rule outside the module: after N consecutive held busy ticks with load still above target (for example 3 ticks), scale out anyway by one step and page the on-call. Respect your max-capacity and step limits as today.
• Oscillation. Repeated holds on the agreement check usually point at a retry storm or a load balancer problem; route them to the owning team rather than adding capacity.
• Feedback. The weekly calibration refresh picks up the outcomes of acted scale-outs. Overrides by the bounded-hold rule are worth reviewing: if they were needed, they belong in the justified set.

Tuning and expected workload

• TARGET_REQUESTS_PER_INSTANCE = 80 is your existing target.
• LOOKBACK_MINUTES = 15 and LOAD_PATTERN_DENSITY_THRESHOLD = 0.25 together decide how much history is required (see the warning above). Lengthen the lookback and you react later but hold less after gaps.
• α = 0.10 (unusualness) decides how lopsided a window may be compared with past justified scale-outs. Raising it holds more blips and also more early-surge ticks.
• Expected holds: roughly every tick where today's cooldown would have suppressed or reversed a scale, plus every tick in the minutes after a metrics gap. Measure in shadow mode; see estimate impact and choose parameters.

Limits, risks and compliance

• Scale-in ignores the calendar. Fifteen quiet minutes before a scheduled peak (a batch job, a marketing send, Monday morning) will scale in. Keep scheduled scaling and minimum-capacity floors outside this module.
• Holds during real surges need the bounded-hold rule above.
• Synthetic data. Load samples and calibration shares are invented.

Wiring it in

1. At autoscaler start, build the module once: scaling_module = build().
2. Weekly, refresh the calibration tuples from the autoscaler's action log joined with utilisation.
3. Each evaluation tick, run the range query for the last fifteen minutes, name the event from the latest sample, build the evidence and decide:

from grounded_decision.examples.cloud_autoscaling import build, load_window_evidence

event = Event("busy", {"service": service, "latest_requests_per_instance": latest})
evidence = load_window_evidence(samples, event_name=event.name)
response = handle_request(
    scaling_module,
    {
        "state": None,  # every tick is independent
        "event": event.to_dict(),
        "evidence": evidence.to_dict(),
    },
)
append_to_audit_log(response)
if response["verdict"] == "act":
    call_scaling_api(service, response["action"])  # scale_out / scale_in
else:
    emit_metric("autoscaler.hold", reason=response["reason"])

append_to_audit_log, call_scaling_api and emit_metric are your own functions.

1. If you want a per-service cooldown state, persist response["extras"]["state"] per service and pass it back instead of None.

The policy is the library's two-path helper: busy proposes scale_out, quiet proposes scale_in.

Run it yourself

uv run grounded-decision examples cloud_autoscaling

CLI label	Meaning here
fire, ACT	act: the proposed action is carried out
HOLD, hold_for_human, ABSTAIN	hold: nothing is carried out; record verdict abstain
strength dλ², "KS"	evidence strength and its 1.05 line
"flicker band", p*, FLICKER	the knife-edge window and its cut-off
bag	the calibration set
cursor, extras['state']	the saved state (not used here: ticks are independent)

The naive-versus-grounded counts are in the summary field of --json output.

Further reading

• Act, abstain, degrade
• The six checks
• Troubleshoot: why did it abstain?, Operate and monitor
• Theory: recovering a label from related cases, knife-edge densities, coverage for unfamiliar cases
• Glossary