Maintain 99.9% Uptime — Site Reliability Engineering

By the end of this page, you will understand how SREs define SLOs/SLIs, set up monitoring with Prometheus and Grafana, and build self-healing infrastructure — and how AI agents can automate operations.

Operations & Monitoring — The 2-Minute Overview

Think about the last time you stayed in a hotel. You didn't see the building management system — monitoring room temperatures, water pressure, elevator status, and fire alarms 24/7. You just had hot water and working Wi-Fi. But somebody had to define "acceptable" (water between 38-42°C), set up monitoring (sensors in every room), and build automatic responses (pressure drops → backup pump activates). That building management is SRE.

You Already Know SRE — You Just Don't Know It Yet

You've been an SRE every time you maintained an aquarium.

🐠 The Aquarium Analogy

Step 1 — Define SLOs: Water temperature: 24-26°C. pH: 7.0-7.5. Oxygen: >6 mg/L.

🔗 SRE Layer: ① SLOs/SLIs — Define what "healthy" means for each service.

Step 2 — Set up monitoring: Thermometer (always visible), pH test kit (weekly), oxygen meter (daily).

🔗 SRE Layer: ② MONITORING — Prometheus collects metrics, Grafana displays dashboards, alerts fire when thresholds are breached.

Step 3 — Self-healing: Heater automatically maintains temperature. Filter runs continuously. Auto-feeder dispenses food.

🔗 SRE Layer: ③ SELF-HEALING — Auto-scaling on high CPU, auto-restart on container crash, circuit breakers on dependency failure.

The Complete Mapping

You just learned SRE without opening a terminal.

The 5 Pillars of Site Reliability Engineering

1. SLOs and SLIs

SLOs are promises to users. SLIs measure if you're keeping them. Error budgets tell you how much room you have.

Define an SLO for each critical user journey. Measure it with an SLI. Track your error budget — the amount of allowed unreliability.

2. Monitoring with Prometheus & Grafana

If you're not measuring it, you're guessing. If you're measuring the wrong thing, you're confidently wrong.

Prometheus scrapes metrics (request count, latency, error rate, resource utilization). Grafana visualizes them in dashboards. Together, they answer: "Is the system healthy right now?" and "Is it trending toward unhealthy?"

3. Alerting Strategy

Every alert must be actionable. If an alert doesn't require human action, it's noise — remove it.

Good alerts: "p99 latency > 500ms for 5 minutes → investigate." Bad alerts: "CPU at 60%" — so what? Alert fatigue is real: too many alerts → humans ignore all of them → real incidents go unnoticed.

4. Self-Healing Infrastructure

The best incident is the one that resolves itself before a human notices.

Auto-scaling: add instances when CPU > 70%. Auto-restart: container crashes → orchestrator restarts within 30 seconds. Circuit breakers: dependency fails → serve cached/default response. Health checks: load balancer stops routing to unhealthy instances.

5. Incident Response & On-Call

On-call isn't about being awake at 3am. It's about having runbooks that make 3am incidents solvable in 10 minutes.

On-call rotation ensures someone is always responsible. Runbooks provide step-by-step instructions for known failure modes. Escalation paths define who to call when runbooks aren't enough.

The Complete Mapping

Master these 5 pillars, master reliability.

Try It Yourself — A Starter Prompt for SRE

This prompt gives you a working starting point. For the complete prompt — with Prometheus configs, Grafana dashboard JSON, and runbook templates — see the full course chapter →.

You are a Senior SRE with experience in Prometheus, Grafana, and self-healing infrastructure.

I need an SRE plan for:

{{PASTE YOUR SYSTEM ARCHITECTURE AND RELIABILITY REQUIREMENTS}}

Cover these 5 areas:

1. SLOs — Define SLOs for the 3 most critical user journeys. Include error budgets.
2. MONITORING — Design the Prometheus metrics and Grafana dashboards needed.
3. ALERTING — Define alerts with severity, threshold, and required action.
4. SELF-HEALING — Design auto-recovery mechanisms for the 3 most likely failure modes.
5. ON-CALL — Define the on-call rotation structure and escalation paths.

For each area, provide: the design and a brief justification.

Format as a structured document with tables.

What This Prompt Covers vs. What It Misses

The Lite Prompt gets you to ~60% quality. Good enough to understand SRE. Not good enough to maintain 99.9% uptime.

Real-World Example: SRE for a Messaging Platform

The Requirement

"Design SRE for a messaging platform: real-time messages, online presence, and push notifications. Target: 99.95% uptime. p99 message delivery < 500ms."

Lite Prompt Output

① SLOs: Message delivery: 99.95% success, p99 < 500ms. Presence: 99.9% accuracy. Push: 99% delivered within 30s.

② Monitoring: Prometheus metrics for message latency, delivery rate, presence accuracy. Grafana dashboard per service.

③ Alerting: P0: delivery rate < 99% for 5 min. P1: p99 > 500ms for 10 min. P2: push delay > 60s.

④ Self-Healing: Auto-scale message service on high load. Auto-restart presence service on crash.

⑤ On-Call: Weekly rotation. Escalation: L1 → L2 → engineering lead.

What an SRE Manager Would Catch

The pattern: The Lite Prompt asks "what's the SRE plan?" The full course asks "what's the plan, what's the error budget policy, and what happens at 3am?"

Ready to Maintain 99.9% Uptime?

• ✅ The complete prompt with Prometheus configs, Grafana dashboards, and runbook templates
• ✅ An AI agent that monitors and self-heals infrastructure
• ✅ Assessment + coding challenges to verify you can operate, not just design

Enroll in the Fresh Graduate AI SDLC Course →

Go from "I understand SRE" to "I can maintain 99.9% uptime for a production system."