Alerting Runbook

Purpose

This runbook documents the alerting architecture — what systems send alerts, where alerts go, who receives them, and how to respond. It serves as the authoritative reference for alert routing and escalation, and as compliance evidence that critical systems are monitored.

---

Alert Routing Overview

Alert Sources
├── Azure Monitor (VM health, backup failures, resource health)
├── AWS CloudWatch (EC2, Lambda, billing anomalies)
├── Arctic Wolf SIEM (security events, threat detection)
├── Palo Alto Firewall (threat prevention, system events)
├── GitHub Actions (pipeline failures, security scan failures)
└── ops-automation (weekly HIPAA audit, cost report, backup verification)
         │
         ▼
    Notification Channels
    ├── IT Admin email (primary — all critical alerts)
    ├── Leadership email (P1 incidents, HIPAA breach notifications)
    └── SES group mailboxes (vulnerability notifications by device group)

---

Alert Severity & Response Times

Severity	Description	Response Time	Who
P1 — Critical	Active breach, ransomware, complete outage, HIPAA breach	Immediate (24/7)	IT Admin + Leadership
P2 — High	Compromised account, significant service degradation, backup failure	Within 1 hour	IT Admin
P3 — Medium	Single server down, failed CI/CD deploy, compliance finding	Within 4 hours	IT Admin
P4 — Low	Minor warnings, informational findings	Next business day	IT Admin

---

Alert Sources

Azure Monitor

Azure Monitor alerts are configured in the azure-infra and azure-dde-infra Terraform repos. Action groups route alerts to the IT admin email.

Alert	Severity	Trigger Condition
VM availability	P2	VM heartbeat lost > 5 minutes
Backup job failure	P2	Recovery Services Vault job status = Failed
Backup job missed	P2	No successful backup within schedule window
High CPU (> 90%)	P3	VM CPU average > 90% for 15 minutes
Storage account near capacity	P3	Storage account > 90% capacity
Resource health degraded	P3	Azure resource health status = Degraded

Where to check: Azure portal → Monitor → Alerts

AWS CloudWatch

CloudWatch alarms are configured in the aws-infra Terraform repo. SNS topics route to the IT admin email.

Alert	Severity	Trigger Condition
EC2 status check failure	P2	EC2 system or instance status check failing
Lambda function errors	P3	Lambda error rate > threshold
S3 bucket size anomaly	P4	Unexpected spike in S3 storage consumption
Billing alert	P3	Monthly spend exceeds defined threshold

Where to check: AWS console → CloudWatch → Alarms

Arctic Wolf SIEM

Arctic Wolf generates alerts based on threat detection, behavioral analysis, and correlation rules. Critical security alerts generate email notifications to IT admin.

Alert Category	Examples	Severity
Ransomware indicators	File encryption activity, shadow copy deletion	P1
Brute force	Multiple failed authentication attempts	P2
Lateral movement	Unusual internal network scanning	P1
Exfiltration	Unusual outbound data volume	P1
Vulnerability	Critical CVE detected on production asset	P2
Policy violation	User accessing restricted resources	P3

Where to check: Arctic Wolf portal → Alerts

Important: Arctic Wolf alerts are the primary source for active threat detection. Check the Arctic Wolf portal at least weekly, and immediately when a P1/P2 email arrives.

Palo Alto Firewall

The Palo Alto firewalls generate system-level alerts for hardware events, HA failover, disk space, and software update failures. These are configured to send syslog to Arctic Wolf.

Alert	Severity	Description
HA failover	P1	Firewall HA pair failed over to secondary
Disk space > 80%	P2	Panorama or firewall log disk approaching limit
Software update failure	P3	PAN-OS or signature update failed
Admin login failure	P2	Multiple failed admin login attempts
Certificate expiry	P3	SSL/TLS certificate approaching expiration

Where to check: Firewall management UI → Monitor → Logs → System; or Arctic Wolf portal (if syslog forwarding is configured)

GitHub Actions (CI/CD Failures)

GitHub Actions sends email notifications to the repository owner when workflows fail. Pull requests with failing security scans cannot be merged — the PR itself blocks until the check passes.

Failure Type	Severity	Action
Checkov HIPAA scan failure	P3	Review PR — a compliance issue was introduced
Trivy scan failure	P3	Review PR — a vulnerability was detected
Terraform apply failure	P2	Infrastructure deployment failed — review and fix
ops-automation workflow failure	P3	HIPAA audit or cost report failed to run

Where to check: GitHub → Actions tab in the relevant repository

---

Weekly Automated Reports (ops-automation)

The ops-automation repository runs scheduled workflows that deliver summary reports via email. These are not real-time alerts — they are regular-cadence operational summaries.

Report	Schedule	Contents	Severity Tier
HIPAA Compliance Audit	Weekly, Monday 08:00 UTC	Azure Policy compliance, AWS Security Hub findings, GitHub security config, Checkov results	Informational (escalate P2 on critical new findings)
Monthly Cost Report	1st of each month	Azure + AWS costs, month-over-month change, waste items, Advisor recommendations	Informational
Backup Verification (planned)	Weekly	Recovery Vault job status, Veeam replication health	P2 on failure
Vulnerability Notification	On CSV upload	Arctic Wolf vulnerability findings by device group	P2 for critical/high CVEs

If a weekly HIPAA audit email reveals new critical findings (red/failed controls that were previously passing), treat this as a P3 incident and investigate within 4 hours.

---

Escalation Paths

IT Admin Is Unavailable

If the primary IT admin is unavailable during a P1 incident, leadership should:

1. Engage the managed services or IR retainer if one exists
2. Contact the relevant cloud vendor support (Azure or AWS) for platform-level issues
3. Document all actions taken for post-incident review

HIPAA Breach Notification

If an incident involves a potential HIPAA breach (unauthorized access to ePHI), follow the breach notification requirements:

• Notify leadership immediately (P1 response)
• Document the discovery time, scope, and affected data
• HIPAA requires notification to HHS and affected individuals within 60 days of discovery
• Engage legal counsel before making breach notifications
• See Incident Response for detailed procedure

---

Alert Maintenance

Adding a New Alert

1. Define the trigger condition and severity
2. Create the alert in Terraform (Azure Monitor action group, AWS SNS/CloudWatch alarm)
3. Test the alert by simulating the trigger condition in dev
4. Document the alert in this runbook

Reviewing Alert Thresholds

Alert thresholds should be reviewed quarterly (align with DR test schedule):

• Identify alerts that are too noisy (P4 alerts filling inbox)
• Identify gaps — systems with no alerting coverage
• Adjust thresholds based on operational experience

---

Compliance Relevance

Requirement	Control	How Alerting Addresses It
HIPAA §164.308(a)(1)(ii)(C) — Information system activity review	Monitor for security events	Arctic Wolf SIEM with alerting provides ongoing monitoring
HIPAA §164.308(a)(5)(ii)(C) — Log-in monitoring	Monitor log-in attempts	Entra ID sign-in alerts, Palo Alto auth logs
HIPAA §164.308(a)(6)(ii) — Incident reporting	Detect and respond to incidents	Alert routing ensures timely discovery
SOC2 CC7.2 — Monitoring	System monitoring activities	Comprehensive alert coverage across all platforms

---

Document History

Date	Change	Author
March 2026	Initial draft — alert sources, routing, severity levels, escalation paths, compliance mapping	Rory

Related Documentation

• Incident Response
• Logging Architecture
• Vulnerability Management
• Palo Alto Firewall Overview