Azure Monitor Alerts — Operations Runbook

Scope: PROD tenant d477c9f8 (ciriusgroup.com) — infrastructure health alerting only LAW: cirius-logging-law-central (ID: 5d76d1f2, rg-logging-logs, logging subscription) Not in scope: Security events — those go through the 17-agent pipeline to SecOps. Azure Monitor here covers infrastructure health: VM down, disk full, backup failure, quota exhaustion.

1. Azure Monitor Architecture at Cirius

Signal Flow

Azure Resources (VMs, RSVs, Container Apps, Storage, Key Vault, Azure OpenAI)
       │
       ├── Platform Metrics (1-min granularity, automatic, no config required)
       │       → Azure Monitor Metrics Store (93-day hot retention)
       │       → Metric Alert Rules evaluate against thresholds
       │
       ├── Resource Logs / Diagnostic Settings
       │       → LAW: cirius-logging-law-central (5d76d1f2)
       │       → Tables: AzureDiagnostics, AzureActivity, ContainerAppConsoleLogs, etc.
       │       → Scheduled Query Rules (Log Alert Rules) run KQL over these tables
       │
       └── Activity Log (subscription-level control plane events)
               → LAW via diagnostic settings on each subscription
               → Table: AzureActivity

Alert Rule fires
       │
       └── Action Group
               ├── Email: rgarshol@gmail.com
               └── Webhook (optional): Telegram bot endpoint

What Goes Where

Signal Type	Transport	Destination	Latency
VM CPU / memory / disk metrics	Platform metrics → metric alert	Action group	~1 min
Backup job status	AzureDiagnostics in LAW	Scheduled query rule	~5-15 min
Container App restarts	ContainerAppSystemLogs in LAW	Scheduled query rule	~5 min
Key Vault throttle / availability	Platform metrics	Metric alert	~1 min
Storage availability	Platform metrics	Metric alert	~1 min
LAW ingestion cap hit	Platform metrics on LAW	Metric alert	~5 min
Azure OpenAI 429 errors	Platform metrics	Metric alert	See azure-openai-operations.md

Division of Labour — Azure Monitor vs Agent Pipeline

Azure Monitor is infrastructure health. The 17-agent security monitoring pipeline is security events. Do not route security alerts through Azure Monitor.

Alert Category	System
VM down, disk full, high CPU	Azure Monitor
Backup failure	Azure Monitor (also covered by operational agent)
Key Vault unavailable	Azure Monitor
Suspicious login, impossible travel	17-agent pipeline → SecOps
New persistence (scheduled task, service)	17-agent pipeline → SecOps
Log cleared (EventID 1102)	17-agent pipeline → SecOps

Arctic Wolf monitors Azure via its connector and receives its own alert stream — Azure Monitor action groups do not forward to Arctic Wolf. Arctic Wolf has independent visibility.

2. Action Groups

What Action Groups Exist

Cirius uses a single central action group for infrastructure alerts in PROD:

Action Group	Resource Group	Subscription	Purpose
`ag-infra-alerts-prod`	`rg-logging-logs`	Logging	All infra health alerts

One action group is enough for current scale. If alert volume grows, split by severity (critical vs warning) with different receiver lists.

Create Action Group in Terraform

hcl

resource "azurerm_monitor_action_group" "infra_alerts_prod" {
  name                = "ag-infra-alerts-prod"
  resource_group_name = "rg-logging-logs"
  short_name          = "InfraAlert"

  email_receiver {
    name          = "rory"
    email_address = "rgarshol@gmail.com"
  }

  # Optional: Telegram via Logic App or custom webhook endpoint
  webhook_receiver {
    name        = "telegram"
    service_uri = var.telegram_webhook_url   # stored in Key Vault, passed via TF var
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Create Action Group in Portal (one-off, then import to Terraform)

Portal → Monitor → Alerts → Action groups → Create
Subscription: Logging | Resource group: rg-logging-logs
Name: ag-infra-alerts-prod | Display: InfraAlert
Notifications tab → Email/SMS/Push/Voice → Email: rgarshol@gmail.com
(Optional) Actions tab → Webhook → paste Telegram endpoint URL
Review + Create

After portal creation, import before writing Terraform:

bash

terraform import azurerm_monitor_action_group.infra_alerts_prod \
  /subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/microsoft.insights/actionGroups/ag-infra-alerts-prod

Test an Action Group

Portal:

Monitor → Alerts → Action groups → select ag-infra-alerts-prod → Test action group
Select notification type (Email) → Test
Check inbox — arrives within ~2 minutes

CLI:

bash

az monitor action-group test \
  --name ag-infra-alerts-prod \
  --resource-group rg-logging-logs \
  --alert-type email

You will receive a test email from azure-noreply@microsoft.com with subject [Test Notification]. If email doesn't arrive in 5 minutes, check spam and confirm the action group was saved with the correct address.

3. Metric Alerts

Metric alerts evaluate platform metrics against a threshold. No LAW required — data comes directly from the metrics store. Evaluation frequency: 1–5 minutes. Good for: CPU, memory, disk, storage availability, Key Vault throttling.

How to Create — Terraform (azurerm_monitor_metric_alert)

hcl

# VM CPU > 90% for 15 minutes
resource "azurerm_monitor_metric_alert" "vm_cpu_high" {
  name                = "alert-vm-cpu-high-prod"
  resource_group_name = "rg-logging-logs"
  scopes              = [azurerm_windows_virtual_machine.example.id]
  description         = "VM CPU above 90% for 15 minutes"
  severity            = 2   # 0=Critical 1=Error 2=Warning 3=Informational 4=Verbose
  enabled             = true
  frequency           = "PT5M"   # evaluate every 5 min
  window_size         = "PT15M"  # lookback window

  criteria {
    metric_namespace = "Microsoft.Compute/virtualMachines"
    metric_name      = "Percentage CPU"
    aggregation      = "Average"
    operator         = "GreaterThan"
    threshold        = 90
  }

  action {
    action_group_id = azurerm_monitor_action_group.infra_alerts_prod.id
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

How to Create — Portal

Portal → Monitor → Alerts → Create → Alert rule
Scope → select target resource (VM, storage account, etc.)
Condition → select signal → set aggregation, operator, threshold, lookback period
Actions → select ag-infra-alerts-prod
Details → name, resource group (rg-logging-logs), severity
Review + Create
After portal creation, import to Terraform before adding more config.

Common Metric Alert Thresholds

Resource	Metric Name	Namespace	Aggregation	Op	Threshold	Window
VM	Percentage CPU	`Microsoft.Compute/virtualMachines`	Average	>	90	PT15M
VM	Available Memory Bytes	`Microsoft.Compute/virtualMachines`	Average	<	524288000 (500 MB)	PT15M
VM	OS Disk Free Space %	`Microsoft.Compute/virtualMachines`	Average	<	10	PT5M
Storage Account	Availability	`Microsoft.Storage/storageAccounts`	Average	<	99.9	PT5M
Key Vault	ServiceApiResult (Throttled)	`Microsoft.KeyVault/vaults`	Count	>	10	PT5M
Key Vault	Availability	`Microsoft.KeyVault/vaults`	Average	<	100	PT5M
LAW	OverQuotaInGB (ingestion cap)	`Microsoft.OperationalInsights/workspaces`	Total	>	0	PT1H
Container App	RestartCount	`Microsoft.App/containerApps`	Total	>	3	PT1H
Container App	CpuUsage	`Microsoft.App/containerApps`	Average	>	80	PT10M

Memory note: Azure VMs report Available Memory Bytes — you are alerting on what is free, not what is used. A threshold of 524288000 = 500 MB free. Adjust per VM RAM size.

VM disk note: OS Disk Free Space % requires the Azure Monitor Agent (AMA) extension with the Disk performance counter enabled. If the metric shows no data, check that AMA is installed and the DCR includes the disk performance counter.

4. Log-Based Alerts (Scheduled Query Rules)

When to Use vs Metric Alert

Use a metric alert when the signal is a numeric metric with native Azure support (CPU, memory, storage availability). Fast, cheap, no LAW dependency.

Use a log alert (Scheduled Query Rule) when:

Signal only exists in logs (backup job status, restart reasons, custom app logs)
You need to correlate multiple fields (e.g., backup job status AND vault name AND job type)
You want KQL aggregation or windowing more complex than a simple threshold
The metric doesn't exist natively (backup failures have no platform metric)

Minimum evaluation frequency: 5 minutes (PT5M). The resource type azurerm_monitor_scheduled_query_rules_alert_v2 supports 1-min frequency only on certain workspaces. Use PT5M as the safe default.

Log alert latency: 5–15 minutes behind real time. Do not use for P1 hard-realtime signals.

How to Create — Terraform (azurerm_monitor_scheduled_query_rules_alert_v2)

hcl

# Backup job failure — KQL over AzureDiagnostics
resource "azurerm_monitor_scheduled_query_rules_alert_v2" "backup_job_failed" {
  name                = "alert-rsv-backup-failure-prod"
  resource_group_name = "rg-logging-logs"
  location            = "eastus"
  description         = "RSV backup job failed in last 24 hours"
  enabled             = true
  severity            = 1   # Error
  evaluation_frequency = "PT5M"
  window_duration      = "PT60M"
  scopes               = ["/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/microsoft.operationalinsights/workspaces/cirius-logging-law-central"]

  criteria {
    query = <<-KQL
      AzureDiagnostics
      | where Category == "AzureBackupReport"
      | where OperationName == "Job"
      | where JobStatus_s == "Failed"
      | where TimeGenerated >= ago(1h)
      | summarize FailedJobs = count() by JobType_s, BackupItemUniqueId_s, Resource
    KQL

    time_aggregation_method = "Count"
    threshold               = 0
    operator                = "GreaterThan"

    failing_periods {
      minimum_failing_periods_to_trigger_alert = 1
      number_of_evaluation_periods             = 1
    }
  }

  action {
    action_groups = [azurerm_monitor_action_group.infra_alerts_prod.id]
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Alert Deduplication

Scheduled Query Rules in Azure Monitor have built-in alert deduplication via the alert state — the same alert rule will not re-fire while it is in a Fired state unless the condition resolves and fires again. This prevents alert spam on persistent conditions.

For additional deduplication on flapping conditions, use auto-mitigate = false combined with alert processing rules (see Section 7).

5. Recommended Alert Baseline for Cirius

All alert rules should be defined in the azure-infra Terraform repo under modules/monitor-alerts/. The LAW resource ID for scopes in log alert rules:

/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/microsoft.operationalinsights/workspaces/cirius-logging-law-central

VM Alerts

CPU > 90% for 15 minutes (metric alert, severity 2)

Metric: Percentage CPU, aggregation: Average, threshold: 90, window: PT15M
Scope: all production VMs (set per-VM or use a resource group scope)

Available Memory < 500 MB (metric alert, severity 2)

Metric: Available Memory Bytes, aggregation: Average, threshold: 524288000 (bytes)
Window: PT15M, frequency: PT5M
Requires: AMA extension installed

OS Disk free < 10% (metric alert, severity 1)

Metric: OS Disk Free Space %, aggregation: Average, threshold: 10
Window: PT5M, frequency: PT5M
Requires: AMA + disk performance counter in DCR

Container App (ca-soc-prod)

Restarts > 3 in 1 hour (metric alert, severity 1)

Metric: RestartCount, aggregation: Total, threshold: 3, window: PT1H
Scope: ca-soc-prod resource ID in rg-logging-logs

hcl

resource "azurerm_monitor_metric_alert" "ca_soc_restarts" {
  name                = "alert-ca-soc-prod-restarts"
  resource_group_name = "rg-logging-logs"
  scopes              = ["/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.App/containerApps/ca-soc-prod"]
  description         = "ca-soc-prod restarted more than 3 times in an hour"
  severity            = 1
  frequency           = "PT5M"
  window_size         = "PT1H"

  criteria {
    metric_namespace = "Microsoft.App/containerApps"
    metric_name      = "RestartCount"
    aggregation      = "Total"
    operator         = "GreaterThan"
    threshold        = 3
  }

  action {
    action_group_id = azurerm_monitor_action_group.infra_alerts_prod.id
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

CPU > 80% (metric alert, severity 2)

Metric: CpuUsage, aggregation: Average, threshold: 80, window: PT10M

Recovery Services Vault — Backup Job Failed

KQL log alert over AzureDiagnostics. Requires diagnostic settings on the RSV to forward to LAW. If AzureDiagnostics has no backup data, check RSV → Diagnostic settings → confirm the AzureBackupReport category is enabled to cirius-logging-law-central.

kql

// Fires when any RSV backup job fails in the last hour
AzureDiagnostics
| where Category == "AzureBackupReport"
| where OperationName == "Job"
| where JobStatus_s == "Failed"
| where TimeGenerated >= ago(1h)
| project TimeGenerated, Resource, JobType_s, BackupItemFriendlyName_s, JobFailureCode_s

Severity: 1 (Error), evaluation: PT5M, window: PT1H, threshold: count > 0.

Storage Account — Availability < 99.9%

hcl

resource "azurerm_monitor_metric_alert" "storage_availability" {
  name                = "alert-storage-availability-prod"
  resource_group_name = "rg-logging-logs"
  scopes              = [azurerm_storage_account.example.id]
  description         = "Storage account availability dropped below 99.9%"
  severity            = 1
  frequency           = "PT5M"
  window_size         = "PT15M"

  criteria {
    metric_namespace = "Microsoft.Storage/storageAccounts"
    metric_name      = "Availability"
    aggregation      = "Average"
    operator         = "LessThan"
    threshold        = 99.9
  }

  action {
    action_group_id = azurerm_monitor_action_group.infra_alerts_prod.id
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Key Vault — Throttled Requests > 10 in 5 min

hcl

resource "azurerm_monitor_metric_alert" "keyvault_throttle" {
  name                = "alert-kv-throttle-prod"
  resource_group_name = "rg-logging-logs"
  scopes              = ["/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.KeyVault/vaults/cirius-openai-kv-prod"]
  description         = "Key Vault throttling: >10 throttled requests in 5 minutes"
  severity            = 2
  frequency           = "PT1M"
  window_size         = "PT5M"

  criteria {
    metric_namespace = "Microsoft.KeyVault/vaults"
    metric_name      = "ServiceApiResult"
    aggregation      = "Count"
    operator         = "GreaterThan"
    threshold        = 10

    dimension {
      name     = "StatusCodeClass"
      operator = "Include"
      values   = ["4xx"]
    }
  }

  action {
    action_group_id = azurerm_monitor_action_group.infra_alerts_prod.id
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Key Vault vault unavailable (metric alert, severity 0 / Critical)

Metric: Availability, aggregation: Average, threshold: < 100, window: PT5M
A Key Vault below 100% availability is an anomaly worth an immediate page.

Log Analytics Workspace — Daily Ingestion Cap Hit

This is a critical alert. If LAW hits its daily data cap, it stops ingesting logs. The 17-agent security monitoring pipeline goes blind. Do not set a hard cap (blocked by CLAUDE.md principle). Instead, alert at the threshold so you can investigate before it would be a problem.

hcl

resource "azurerm_monitor_metric_alert" "law_data_cap" {
  name                = "alert-law-ingestion-cap-prod"
  resource_group_name = "rg-logging-logs"
  scopes              = ["/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/microsoft.operationalinsights/workspaces/cirius-logging-law-central"]
  description         = "LAW daily ingestion cap reached — log ingestion has stopped"
  severity            = 0   # Critical
  frequency           = "PT5M"
  window_size         = "PT1H"

  criteria {
    metric_namespace = "Microsoft.OperationalInsights/workspaces"
    metric_name      = "OverQuotaInGB"
    aggregation      = "Total"
    operator         = "GreaterThan"
    threshold        = 0
  }

  action {
    action_group_id = azurerm_monitor_action_group.infra_alerts_prod.id
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

When this fires: immediately check which tables are driving ingestion (Section 9, KQL queries). Check for a runaway log source. Do not raise the cap without understanding the cause.

Azure OpenAI — Sustained 429 Errors

Cross-reference: azure-openai-operations.md — the alert rule and response steps are documented there. Ensure the action group ag-infra-alerts-prod is wired to that alert rule as well.

6. Alert Management

Viewing Fired Alerts in Portal

Portal → Monitor → Alerts → Alert instances

Filter by:

Subscription: Logging (action groups live here; filter by scope for VM-specific alerts)
Time range: last 24 hours
State: Fired / Acknowledged / Closed

Each alert instance shows: alert rule name, fired time, severity, target resource, condition details (what threshold was breached and the actual value).

Acknowledging an Alert

Acknowledging signals that someone has seen the alert and is investigating. It does not resolve the underlying condition.

Portal: click the alert instance → Change state → Acknowledged → add a comment.

CLI:

bash

# List active alert instances
az monitor alert-instances list \
  --resource-group rg-logging-logs \
  --output table

# Acknowledge (requires the alert instance ID from the list output)
az monitor alert-instances update \
  --ids /subscriptions/<sub-id>/providers/Microsoft.AlertsManagement/alertInstances/<instance-id> \
  --state Acknowledged \
  --comment "Investigating elevated CPU on vm-prod-dc01"

Closing an Alert

Close when the condition is resolved and root cause is understood. Always include a note.

Portal: alert instance → Change state → Closed → enter root cause in the comment.

Required comment format:

Root cause: [what caused it]
Resolution: [what was done]
Recurrence risk: [low/medium/high and why]

Example:

Root cause: vm-prod-dc01 CPU spiked during monthly Windows Update installation.
Resolution: Updates completed, CPU returned to baseline.
Recurrence risk: Low — normal monthly patch cycle. Will recur next Patch Tuesday; create maintenance window.

Do not close without a root cause. Unclosed alerts accumulate and make the Alerts blade useless. Unexplained alerts also create compliance gaps — auditors will ask why no root cause was documented.

7. Alert Noise Management

Alert Processing Rules (Suppression)

Alert processing rules let you suppress or modify actions on alerts matching certain criteria without changing the alert rule itself. Use for:

Scheduled maintenance windows
Known noisy alerts pending threshold tuning
Suppressing non-actionable alerts during off-hours

Create a maintenance window alert processing rule (Terraform):

hcl

resource "azurerm_monitor_alert_processing_rule_suppression" "patch_maintenance" {
  name                = "apr-patch-tuesday-maintenance"
  resource_group_name = "rg-logging-logs"
  description         = "Suppress infra alerts during monthly Patch Tuesday window"
  enabled             = true

  scopes = [
    "/subscriptions/<prod-sub-id>",
    "/subscriptions/<logging-sub-id>"
  ]

  condition {
    severity {
      operator = "Equals"
      values   = ["Sev2", "Sev3", "Sev4"]   # suppress Warning and below
    }
  }

  schedule {
    effective_from  = "2026-06-09T22:00:00"   # second Tuesday + maintenance start
    effective_until = "2026-06-10T06:00:00"
    time_zone       = "Pacific Standard Time"

    recurrence {
      monthly {
        days_of_month = [9, 10, 11, 12, 13, 14, 15]   # catches second Tuesday of any month
        # Combine with day-of-week filter in condition if needed
      }
    }
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Portal: Monitor → Alerts → Alert processing rules → Create

Important: Alert processing rules suppress the action (email/webhook). The alert still fires and appears in the Alerts blade — it just doesn't send a notification. This means you can still see it after the maintenance window without losing history.

Reducing False Positives — Threshold Guidance

Problem	Fix
CPU alert fires during every backup job	Increase window from PT15M to PT30M, or exclude backup hours via alert processing rule
Memory alert fires on boot	Add 10-minute delay with `auto_mitigate = true` and a longer window
Backup alert fires for known excluded VMs	Scope alert rules to specific RSVs, not the whole resource group
Container App alert fires during deploy restarts	Create an alert processing rule tied to CM ticket windows

Flapping Alerts

A flapping alert repeatedly fires and resolves. Two fixes:

Widen the evaluation window. Change PT5M window to PT15M or PT30M. The threshold must be sustained over the longer window before firing.
Raise the threshold. If CPU at 91% for 15 min is genuinely not a problem (it resolves on its own), raise the threshold to 95% or reduce from Average to the P90 percentile (requires a log alert with KQL, not a metric alert).

auto_mitigate = true (the default) means Azure automatically closes the alert when the condition drops below threshold. This is correct behavior — do not set it to false unless you specifically want the alert to persist past resolution.

8. Terraform Patterns

Resource ID Reference

The LAW scope string for log alert rules:

/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/microsoft.operationalinsights/workspaces/cirius-logging-law-central

Store this in a local or variable to avoid repetition:

hcl

locals {
  law_id = "/subscriptions/${var.logging_subscription_id}/resourceGroups/rg-logging-logs/providers/microsoft.operationalinsights/workspaces/cirius-logging-law-central"
}

Full azurerm_monitor_metric_alert Pattern

hcl

resource "azurerm_monitor_metric_alert" "example" {
  name                = "alert-<resource>-<condition>-<env>"
  resource_group_name = "rg-logging-logs"
  scopes              = [var.target_resource_id]
  description         = "Human-readable description of what this alert means"
  severity            = 2        # 0=Critical 1=Error 2=Warning 3=Informational
  enabled             = true
  frequency           = "PT5M"   # PT1M, PT5M, PT15M, PT30M, PT1H
  window_size         = "PT15M"  # must be >= frequency; PT5M to PT1D
  auto_mitigate       = true     # auto-close when condition clears

  criteria {
    metric_namespace = "Microsoft.Compute/virtualMachines"   # exact namespace string
    metric_name      = "Percentage CPU"
    aggregation      = "Average"   # Average, Count, Maximum, Minimum, Total
    operator         = "GreaterThan"
    threshold        = 90

    # Optional: filter by dimension (e.g. disk, container name)
    # dimension {
    #   name     = "vmName"
    #   operator = "Include"
    #   values   = ["*"]
    # }
  }

  action {
    action_group_id = azurerm_monitor_action_group.infra_alerts_prod.id
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Full azurerm_monitor_scheduled_query_rules_alert_v2 Pattern

hcl

resource "azurerm_monitor_scheduled_query_rules_alert_v2" "example" {
  name                = "alert-sqr-<condition>-<env>"
  resource_group_name = "rg-logging-logs"
  location            = "eastus"
  description         = "Description"
  enabled             = true
  severity            = 1

  # How often the rule evaluates
  evaluation_frequency = "PT5M"   # minimum is PT5M for most workspace types

  # How far back each evaluation looks
  window_duration = "PT1H"

  # Must be the LAW resource ID for KQL-over-LAW alerts
  scopes = [local.law_id]

  # Optional: mute alert re-notifications for this duration after firing
  mute_actions_duration = "PT1H"

  criteria {
    query = <<-KQL
      AzureDiagnostics
      | where Category == "AzureBackupReport"
      | where OperationName == "Job"
      | where JobStatus_s == "Failed"
      | where TimeGenerated >= ago(1h)
      | summarize FailedJobs = count()
    KQL

    time_aggregation_method = "Count"   # Count, Average, Maximum, Minimum, Total
    threshold               = 0
    operator                = "GreaterThan"

    failing_periods {
      minimum_failing_periods_to_trigger_alert = 1
      number_of_evaluation_periods             = 1
    }
  }

  action {
    action_groups = [azurerm_monitor_action_group.infra_alerts_prod.id]
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Common Errors

Error	Cause	Fix
`window_duration must be >= evaluation_frequency`	Window shorter than eval period	Set window_duration >= evaluation_frequency
`The scope is not supported`	Wrong resource type for metric namespace	Verify namespace matches scope resource type
No data in KQL log alert	Diagnostic settings missing	Enable AzureBackupReport / relevant category on the resource
`Threshold must be > 0 for Count aggregation`	Some aggregations require nonzero	Use `>= 1` instead of `> 0` for count-type queries

9. Checking Alert History

Portal

Monitor → Alerts → Alert instances

Default view shows last 24 hours
Filter by time range (up to 30 days)
Export to CSV for audit documentation

Monitor → Alerts → Alert rules → click a rule → Alert instance history tab

Shows all fires for that specific rule
Includes condition value at time of fire

KQL — Fired Alert Events in AzureActivity

AzureActivity captures when alert rules fire as microsoft.insights/alertrules/activated/action events. Query the LAW:

kql

// All alert activations in the last 7 days
AzureActivity
| where TimeGenerated >= ago(7d)
| where OperationNameValue contains "microsoft.insights/alert"
| where ActivityStatusValue == "Succeeded"
| project TimeGenerated, OperationNameValue, ResourceGroup, Resource, Properties
| sort by TimeGenerated desc

kql

// Alert activations with alert name and description
AzureActivity
| where TimeGenerated >= ago(7d)
| where OperationNameValue == "microsoft.insights/alertrules/activated/action"
| extend AlertName = tostring(parse_json(Properties).alertRuleName)
| extend Description = tostring(parse_json(Properties).description)
| project TimeGenerated, AlertName, Description, ResourceGroup
| sort by TimeGenerated desc

kql

// Count alert fires by rule — useful for identifying noisy rules
AzureActivity
| where TimeGenerated >= ago(30d)
| where OperationNameValue == "microsoft.insights/alertrules/activated/action"
| extend AlertName = tostring(parse_json(Properties).alertRuleName)
| summarize FireCount = count() by AlertName
| sort by FireCount desc

Azure CLI — List Alert Rules

bash

# List all metric alert rules in the logging RG
az monitor metrics alert list \
  --resource-group rg-logging-logs \
  --output table \
  --query "[].{Name:name, Enabled:enabled, Severity:severity, Description:description}"

# List all scheduled query rules in the logging RG
az monitor scheduled-query list \
  --resource-group rg-logging-logs \
  --output table

Azure CLI — Check a Specific Alert's Recent State

bash

# Show the current state and last evaluation of a metric alert
az monitor metrics alert show \
  --name alert-vm-cpu-high-prod \
  --resource-group rg-logging-logs

# Get alert instances (fired alerts) for last 24 hours
az monitor alert-instances list \
  --output table

Quick Reference — Resource IDs and Namespaces

Resource	Namespace
Virtual Machines	`Microsoft.Compute/virtualMachines`
Container Apps	`Microsoft.App/containerApps`
Recovery Services Vaults	`Microsoft.RecoveryServices/vaults`
Storage Accounts	`Microsoft.Storage/storageAccounts`
Key Vault	`Microsoft.KeyVault/vaults`
Log Analytics Workspace	`Microsoft.OperationalInsights/workspaces`
Azure OpenAI / Cognitive Services	`Microsoft.CognitiveServices/accounts`

Severity Number	Label
0	Critical
1	Error
2	Warning
3	Informational
4	Verbose

Frequency Token	Meaning
PT1M	1 minute
PT5M	5 minutes
PT15M	15 minutes
PT30M	30 minutes
PT1H	1 hour

Azure Monitor Alerts — Operations Runbook ​

1. Azure Monitor Architecture at Cirius ​

Signal Flow ​

What Goes Where ​

Division of Labour — Azure Monitor vs Agent Pipeline ​

2. Action Groups ​

What Action Groups Exist ​

Create Action Group in Terraform ​

Create Action Group in Portal (one-off, then import to Terraform) ​

Test an Action Group ​

3. Metric Alerts ​

How to Create — Terraform (azurerm_monitor_metric_alert) ​

How to Create — Portal ​

Common Metric Alert Thresholds ​

4. Log-Based Alerts (Scheduled Query Rules) ​

When to Use vs Metric Alert ​

How to Create — Terraform (azurerm_monitor_scheduled_query_rules_alert_v2) ​

Alert Deduplication ​

5. Recommended Alert Baseline for Cirius ​

VM Alerts ​

Container App (ca-soc-prod) ​

Recovery Services Vault — Backup Job Failed ​

Storage Account — Availability < 99.9% ​

Key Vault — Throttled Requests > 10 in 5 min ​

Log Analytics Workspace — Daily Ingestion Cap Hit ​

Azure OpenAI — Sustained 429 Errors ​

6. Alert Management ​

Viewing Fired Alerts in Portal ​

Acknowledging an Alert ​

Closing an Alert ​

7. Alert Noise Management ​

Alert Processing Rules (Suppression) ​

Reducing False Positives — Threshold Guidance ​

Flapping Alerts ​

8. Terraform Patterns ​

Resource ID Reference ​

Full azurerm_monitor_metric_alert Pattern ​

Full azurerm_monitor_scheduled_query_rules_alert_v2 Pattern ​

Common Errors ​

9. Checking Alert History ​

Portal ​

KQL — Fired Alert Events in AzureActivity ​

Azure CLI — List Alert Rules ​

Azure CLI — Check a Specific Alert's Recent State ​

Quick Reference — Resource IDs and Namespaces ​

Azure Monitor Alerts — Operations Runbook

1. Azure Monitor Architecture at Cirius

Signal Flow

What Goes Where

Division of Labour — Azure Monitor vs Agent Pipeline

2. Action Groups

What Action Groups Exist

Create Action Group in Terraform

Create Action Group in Portal (one-off, then import to Terraform)

Test an Action Group

3. Metric Alerts

How to Create — Terraform (azurerm_monitor_metric_alert)

How to Create — Portal

Common Metric Alert Thresholds

4. Log-Based Alerts (Scheduled Query Rules)

When to Use vs Metric Alert

How to Create — Terraform (azurerm_monitor_scheduled_query_rules_alert_v2)

Alert Deduplication

5. Recommended Alert Baseline for Cirius

VM Alerts

Container App (ca-soc-prod)

Recovery Services Vault — Backup Job Failed

Storage Account — Availability < 99.9%

Key Vault — Throttled Requests > 10 in 5 min

Log Analytics Workspace — Daily Ingestion Cap Hit

Azure OpenAI — Sustained 429 Errors

6. Alert Management

Viewing Fired Alerts in Portal

Acknowledging an Alert

Closing an Alert

7. Alert Noise Management

Alert Processing Rules (Suppression)

Reducing False Positives — Threshold Guidance

Flapping Alerts

8. Terraform Patterns

Resource ID Reference

Full azurerm_monitor_metric_alert Pattern

Full azurerm_monitor_scheduled_query_rules_alert_v2 Pattern

Common Errors

9. Checking Alert History

Portal

KQL — Fired Alert Events in AzureActivity

Azure CLI — List Alert Rules

Azure CLI — Check a Specific Alert's Recent State

Quick Reference — Resource IDs and Namespaces