DCR and AMA Management Runbook

Purpose

This runbook covers the full lifecycle of Data Collection Rules (DCRs) and Azure Monitor Agent (AMA) at Cirius Group. All security event ingestion into cirius-logging-law-central flows through this pipeline. Kill chain detection — EventIDs 4688, 4698, 4104, 7045, 1102, and others — depends on DCRs being correctly configured and associated to every VM.

This runbook is the operational reference for:

Understanding how DCRs work and how they connect to VMs
Listing, inspecting, and validating DCRs in the environment
Creating and updating DCRs in Terraform
Verifying AMA health on Windows and Linux VMs
Troubleshooting data gaps
Adding new VMs to the collection pipeline
Monthly verification checklist

Target workspace: cirius-logging-law-centralWorkspace ID: 5d76d1f2Resource group: rg-logging-logsSubscription: logging subscription

Critical Gotchas — Read Before Touching Anything

These are the most common failure modes. All have been encountered in this environment.

1. Microsoft-SecurityEvent Stream Requires a Sentinel-Enabled Workspace

Using the Microsoft-SecurityEvent stream on a Log Analytics workspace that does not have Microsoft Sentinel enabled returns:

400 InvalidPayload: Data collection rule is invalid

cirius-logging-law-central has Sentinel enabled. This must never change. Do not migrate DCRs that use Microsoft-SecurityEvent to a non-Sentinel workspace.

2. Do Not Use Microsoft-Event as a Workaround

Microsoft-Event works on any LAW, but it delivers events to the Event table with an unparsed RenderedDescription blob rather than structured columns. KQL queries that reference SecurityEvent.CommandLine, SecurityEvent.ProcessName, or SecurityEvent.AccountName will break entirely. The detection agents depend on these columns. Never switch a DCR from Microsoft-SecurityEvent to Microsoft-Event to resolve a 400 error — the correct fix is to ensure the destination workspace has Sentinel enabled.

3. PowerShell 4103/4104 Come from a Different Log Channel

EventIDs 4103 (module logging) and 4104 (script block logging) are written to Microsoft-Windows-PowerShell/Operational, not the Security log. A DCR that only collects from Security!* will never capture PowerShell events, regardless of whether script block logging is enabled on the VM. The PowerShell Operational log must be added as a separate Windows Event Log data source with its own XPath filter.

4. EventID 7045 Lives in the System Log, Not the Security Log

New service installation (EventID 7045) is written to the Windows System log. A DCR that only collects from Security!* will not capture 7045. The System log must be added as a separate data source with its own XPath filter.

5. Never Mix AMA and MMA on the Same VM

Azure Monitor Agent (AMA) and the legacy Log Analytics agent (MMA / Microsoft Monitoring Agent) must not coexist on the same VM. Running both causes duplicate data ingestion and unpredictable collection gaps. If MMA is present on a VM, remove it before deploying AMA. MMA is identified as the OmsAgentForLinux extension (Linux) or MicrosoftMonitoringAgent extension (Windows) in az vm extension list.

Concepts

DCR Components

A Data Collection Rule has three parts:

Data sources — what to collect. For Windows VMs this is Windows Event Logs (with XPath filters), performance counters, or IIS logs. For Linux VMs this is syslog (facility/severity). Each data source references a stream name.

Streams — the schema the data flows through. The two relevant streams:

Microsoft-SecurityEvent — routes to the SecurityEvent table with parsed, structured columns. Requires a Sentinel-enabled workspace.
Microsoft-Event — routes to the Event table with raw RenderedDescription. Works on any LAW but provides no parsed columns.
Microsoft-Syslog — routes to the Syslog table. Used for Linux VMs.

Destinations — where data lands. For Cirius this is always cirius-logging-law-central (workspace ID 5d76d1f2).

DCR Associations

A DCR does nothing until it is associated to one or more VMs. The association is a separate Azure resource (Microsoft.Insights/dataCollectionRuleAssociations). One VM can be associated to multiple DCRs (e.g., a base Windows security DCR and a separate custom log DCR). One DCR can be associated to many VMs.

The association tells AMA on that VM: "download this DCR and collect according to its rules."

Windows Event Log DCR vs Custom Log DCR

Windows Event Log DCR — collects from named Windows log channels (Security, System, Application, Microsoft-Windows-PowerShell/Operational, etc.) using XPath filters. Events land in SecurityEvent (via Microsoft-SecurityEvent stream) or Event (via Microsoft-Event stream). This is what Cirius uses for kill chain detection.

Custom Log DCR — collects text from files on disk (log files, application output) and routes to custom tables (_CL suffix). Not used for Windows event log collection.

Listing and Inspecting DCRs

Azure CLI

List all DCRs in the logging resource group:

bash

az monitor data-collection rule list --resource-group rg-logging-logs --output table

List all DCRs in the subscription:

bash

az monitor data-collection rule list --output table

Show full detail for a specific DCR (JSON):

bash

az monitor data-collection rule show \
  --name <dcr-name> \
  --resource-group rg-logging-logs

List VMs associated to a specific DCR:

bash

az monitor data-collection rule association list \
  --rule-name <dcr-name> \
  --resource-group rg-logging-logs \
  --output table

List all DCR associations for a specific VM:

bash

az monitor data-collection rule association list-by-resource \
  --resource "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Compute/virtualMachines/<vm-name>" \
  --output table

Show the XPath filters active on a DCR:

bash

az monitor data-collection rule show \
  --name <dcr-name> \
  --resource-group rg-logging-logs \
  --query "dataSources.windowsEventLogs[].xPathQueries" \
  --output json

Azure Portal

View DCRs:Azure Portal → Monitor → Data Collection Rules

Filter by resource group: rg-logging-logs

View data sources and XPath filters on a DCR:Monitor → Data Collection Rules → [DCR name] → Data Sources → Windows Event Logs

View associated VMs:Monitor → Data Collection Rules → [DCR name] → Resources

View data flow metrics:Monitor → Data Collection Rules → [DCR name] → Metrics Select metric: Logs Ingestion Bytes per Min or Logs Ingestion Events per Min

Creating and Updating DCRs

Terraform: Main Kill Chain DCR

All DCR changes go through the azure-infra repo. The pattern below is the complete Terraform block for the kill chain DCR covering Security, System, PowerShell Operational, and Application logs.

hcl

resource "azurerm_monitor_data_collection_rule" "kill_chain" {
  name                = "cirius-kill-chain-dcr"
  resource_group_name = "rg-logging-logs"
  location            = "eastus"

  destinations {
    log_analytics {
      workspace_resource_id = azurerm_log_analytics_workspace.central.id
      name                  = "law-central"
    }
  }

  data_flow {
    streams      = ["Microsoft-SecurityEvent"]
    destinations = ["law-central"]
  }

  data_sources {
    # Security log — logon, process creation, privilege, persistence, defense evasion
    windows_event_log {
      name    = "security-log"
      streams = ["Microsoft-SecurityEvent"]
      x_path_queries = [
        "Security!*[System[(EventID=4624 or EventID=4625 or EventID=4648 or EventID=4688 or EventID=4698 or EventID=4720 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4768 or EventID=4769 or EventID=4776 or EventID=1102)]]"
      ]
    }

    # System log — new service installation (EventID 7045)
    # 7045 is NOT in the Security log. Must be a separate data source.
    windows_event_log {
      name    = "system-log"
      streams = ["Microsoft-SecurityEvent"]
      x_path_queries = [
        "System!*[System[(EventID=7045)]]"
      ]
    }

    # PowerShell Operational log — module logging (4103) and script block logging (4104)
    # These events are NOT in the Security log. Must be a separate data source.
    windows_event_log {
      name    = "powershell-operational"
      streams = ["Microsoft-SecurityEvent"]
      x_path_queries = [
        "Microsoft-Windows-PowerShell/Operational!*[System[(EventID=4103 or EventID=4104)]]"
      ]
    }

    # Application log — application errors and relevant application events
    windows_event_log {
      name    = "application-log"
      streams = ["Microsoft-SecurityEvent"]
      x_path_queries = [
        "Application!*[System[(Level=1 or Level=2)]]"
      ]
    }
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

XPath Filter Reference

XPath filters follow the format: <LogChannel>!<XPath expression>

Security log — specific EventIDs:

Security!*[System[(EventID=4624 or EventID=4625 or EventID=4648 or EventID=4688 or EventID=4698 or EventID=4720 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=4768 or EventID=4769 or EventID=4776 or EventID=1102)]]

System log — EventID 7045 only:

System!*[System[(EventID=7045)]]

PowerShell Operational — EventIDs 4103 and 4104:

Microsoft-Windows-PowerShell/Operational!*[System[(EventID=4103 or EventID=4104)]]

Application log — errors and critical only (Level 1 and 2):

Application!*[System[(Level=1 or Level=2)]]

All events from a log channel (use sparingly — high volume):

Security!*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=0)]]

Note on log channel names: Channel names are case-sensitive in the XML query but Azure accepts them case-insensitively in most cases. Use the exact canonical names shown above to avoid ambiguity. Microsoft-Windows-PowerShell/Operational with the forward slash is required — Microsoft-Windows-PowerShell\Operational (backslash) does not work.

Terraform: DCR Association to VMs

Associate the DCR to each VM that should collect events. One association resource per VM.

hcl

resource "azurerm_monitor_data_collection_rule_association" "kill_chain_vm" {
  for_each = toset(var.monitored_vm_ids)

  name                    = "assoc-kill-chain-${basename(each.value)}"
  target_resource_id      = each.value
  data_collection_rule_id = azurerm_monitor_data_collection_rule.kill_chain.id
  description             = "Kill chain DCR association for ${basename(each.value)}"
}

Where var.monitored_vm_ids is a list of full VM resource IDs:

hcl

variable "monitored_vm_ids" {
  description = "Resource IDs of VMs to associate with the kill chain DCR"
  type        = list(string)
}

CLI: Manual DCR Association (Out-of-Band Fix)

Use this when a VM needs immediate association without waiting for a Terraform PR cycle. Follow up with a Terraform PR to bring the association into IaC.

bash

az monitor data-collection rule association create \
  --name "assoc-kill-chain-<vm-name>" \
  --rule-id "/subscriptions/<sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.Insights/dataCollectionRules/cirius-kill-chain-dcr" \
  --resource "/subscriptions/<sub-id>/resourceGroups/<vm-rg>/providers/Microsoft.Compute/virtualMachines/<vm-name>"

Verify the association was created:

bash

az monitor data-collection rule association list-by-resource \
  --resource "/subscriptions/<sub-id>/resourceGroups/<vm-rg>/providers/Microsoft.Compute/virtualMachines/<vm-name>" \
  --output table

Importing Existing DCRs into Terraform State

If a DCR exists in Azure but is not yet managed by Terraform:

bash

terraform import \
  azurerm_monitor_data_collection_rule.kill_chain \
  "/subscriptions/<sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.Insights/dataCollectionRules/cirius-kill-chain-dcr"

For associations:

bash

terraform import \
  "azurerm_monitor_data_collection_rule_association.kill_chain_vm[\"<vm-resource-id>\"]" \
  "/subscriptions/<sub-id>/resourceGroups/<vm-rg>/providers/Microsoft.Compute/virtualMachines/<vm-name>/providers/Microsoft.Insights/dataCollectionRuleAssociations/assoc-kill-chain-<vm-name>"

After import, run terraform plan and confirm zero diff before merging.

AMA Health Checks

Verify AMA Is Installed on a VM

Azure CLI — list extensions on a Windows VM:

bash

az vm extension list \
  --vm-name <vm-name> \
  --resource-group <rg> \
  --output table

Look for AzureMonitorWindowsAgent (Windows) or AzureMonitorLinuxAgent (Linux) in the name column. provisioningState should be Succeeded.

Azure CLI — check extension provisioning state specifically:

bash

az vm extension show \
  --vm-name <vm-name> \
  --resource-group <rg> \
  --name AzureMonitorWindowsAgent \
  --query "provisioningState" \
  --output tsv

Verify AMA Service Is Running

Windows — local PowerShell on the VM:

powershell

Get-Service -Name AzureMonitorAgent | Select-Object Name, Status, StartType

Expected: Status = Running, StartType = Automatic

Windows — remote check via Invoke-AzVMRunCommand:

powershell

Invoke-AzVMRunCommand `
  -ResourceGroupName <rg> `
  -VMName <vm-name> `
  -CommandId RunPowerShellScript `
  -ScriptString "Get-Service -Name AzureMonitorAgent | Select-Object Name, Status, StartType"

Linux — local:

bash

systemctl status azuremonitoragent

Linux — remote via Invoke-AzVMRunCommand:

powershell

Invoke-AzVMRunCommand `
  -ResourceGroupName <rg> `
  -VMName <vm-name> `
  -CommandId RunShellScript `
  -ScriptString "systemctl status azuremonitoragent"

AMA Log Locations

Windows:

C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\

Key files:

CommandExecution.log — extension install and upgrade operations
AMAEventLogs\ — agent operational events including DCR download and upload errors

Linux:

/var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/

Key files:

CommandExecution.log — extension operations
/var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/extension.log

Reading AMA logs on Windows via PowerShell:

powershell

Get-Content "C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\CommandExecution.log" -Tail 100

Reading AMA logs on Linux:

bash

tail -100 /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/extension.log

Restart AMA (When Needed)

Windows:

powershell

Restart-Service -Name AzureMonitorAgent -Force

Linux:

bash

sudo systemctl restart azuremonitoragent

After restart, allow 5 minutes for the agent to re-download its DCR configuration and resume event forwarding.

Troubleshooting — Data Not Flowing

Work through these steps in order. Most gaps are resolved at Step 1 or Step 2.

Step 1 — Verify AMA Service Is Running on the VM

On the affected VM (or via RunCommand):

Windows:

powershell

Get-Service -Name AzureMonitorAgent

If status is Stopped: start it.

powershell

Start-Service -Name AzureMonitorAgent

If it fails to start, check the AMA extension provisioning state in the portal: VM → Extensions + applications → AzureMonitorWindowsAgent

If provisioning state is Failed, reinstall the extension:

bash

az vm extension delete \
  --vm-name <vm-name> \
  --resource-group <rg> \
  --name AzureMonitorWindowsAgent

az vm extension set \
  --vm-name <vm-name> \
  --resource-group <rg> \
  --name AzureMonitorWindowsAgent \
  --publisher Microsoft.Azure.Monitor \
  --enable-auto-upgrade true

Linux:

bash

systemctl status azuremonitoragent
# If dead: sudo systemctl start azuremonitoragent

Step 2 — Check DCR Association Exists for That VM

bash

az monitor data-collection rule association list-by-resource \
  --resource "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Compute/virtualMachines/<vm-name>" \
  --output table

If the kill chain DCR is not listed, add the association:

bash

az monitor data-collection rule association create \
  --name "assoc-kill-chain-<vm-name>" \
  --rule-id "/subscriptions/<sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.Insights/dataCollectionRules/cirius-kill-chain-dcr" \
  --resource "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Compute/virtualMachines/<vm-name>"

After adding the association, restart AMA on the VM to force immediate DCR pickup. Otherwise the agent picks up new associations within 15–30 minutes.

Step 3 — Verify XPath Filter Syntax

Incorrect log channel names or malformed XPath silently result in zero collection. Azure validates syntax at DCR create/update time but does not validate channel names against actual channels on the VM — a typo in the channel name collects nothing.

Common mistakes:

Wrong	Correct
`Security\*`	`Security!*`
`Microsoft-Windows-PowerShell\Operational!*`	`Microsoft-Windows-PowerShell/Operational!*`
`System\*[System[(EventID=7045)]]`	`System!*[System[(EventID=7045)]]`
`security!*` (lowercase)	`Security!*`

Validate the active XPath filters on the DCR:

bash

az monitor data-collection rule show \
  --name cirius-kill-chain-dcr \
  --resource-group rg-logging-logs \
  --query "dataSources.windowsEventLogs[].{name:name, xPath:xPathQueries}" \
  --output json

Confirm that Microsoft-Windows-PowerShell/Operational and System appear as separate data source entries. If only Security is listed, PowerShell events and 7045 will never flow.

Step 4 — Check AMA Logs for Upload Errors

Windows — check for upload errors:

powershell

Get-Content "C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\CommandExecution.log" -Tail 200 | Select-String -Pattern "error|fail|upload|auth" -CaseSensitive:$false

Linux:

bash

sudo grep -i "error\|fail\|upload\|auth" /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent/extension.log | tail -50

Common errors and their fixes:

Error	Cause	Fix
`401 Unauthorized`	VM managed identity not granted LAW contributor	Grant the VM's managed identity `Log Analytics Contributor` on `rg-logging-logs`
`DCR download failed`	Association missing or DCR deleted	Re-add association, verify DCR exists
`400 InvalidPayload`	DCR stream vs workspace mismatch	Confirm Sentinel is enabled on `cirius-logging-law-central`
`Channel not found`	XPath filter references non-existent log channel	Correct the channel name in the DCR

Step 5 — KQL: VMs Sending Heartbeat but No SecurityEvent

Run this in cirius-logging-law-central:

kql

// Machines with AMA alive but no SecurityEvent in last 4 hours
let amaAlive = Heartbeat
    | where TimeGenerated > ago(1h)
    | where OSType == "Windows"
    | summarize LastHeartbeat = max(TimeGenerated) by Computer;
let securtyEventSenders = SecurityEvent
    | where TimeGenerated > ago(4h)
    | summarize LastEvent = max(TimeGenerated) by Computer;
amaAlive
| join kind=leftanti securtyEventSenders on Computer
| project Computer, LastHeartbeat
| sort by Computer asc

VMs in this result are online (AMA is connected to LAW — hence the Heartbeat) but no security events are flowing. This points to either a missing DCR association, incorrect XPath filter, or missing audit policy on the VM.

Variant — check for any specific EventID gap:

kql

// VMs sending 4624 but not 4688 (audit policy applied but CommandLine logging missing)
let logon_senders = SecurityEvent
    | where TimeGenerated > ago(4h)
    | where EventID == 4624
    | summarize by Computer;
let process_senders = SecurityEvent
    | where TimeGenerated > ago(4h)
    | where EventID == 4688
    | summarize by Computer;
logon_senders
| join kind=leftanti process_senders on Computer
| project Computer
| sort by Computer asc

Step 6 — Verify Workspace Is Sentinel-Enabled

If the DCR uses Microsoft-SecurityEvent and returns a 400 error on create/update, confirm Sentinel is enabled on cirius-logging-law-central:

Azure CLI:

bash

az security workspace-setting list --output table

Portal:Microsoft Sentinel → [workspace list]

cirius-logging-law-central should appear. If it does not, Sentinel has been disabled and must be re-enabled before Microsoft-SecurityEvent DCRs will function.

Step 7 — Check DCR Data Flow Metrics in Azure Monitor

Portal path:Monitor → Data Collection Rules → cirius-kill-chain-dcr → Metrics

Select:

Logs Ingestion Bytes per Min — should be non-zero when VMs are active
Logs Ingestion Events per Min — should be non-zero

Filter by Destination = law-central to isolate traffic to cirius-logging-law-central.

Zero metrics on a DCR that has associated VMs with AMA running typically indicates a problem at the workspace or DCR configuration level — not an AMA problem.

KQL Queries for Monitoring

Run all queries against cirius-logging-law-central (workspace ID 5d76d1f2).

Portal: Log Analytics workspaces → cirius-logging-law-central → Logs

CLI:

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "<query>" \
  --output table

Heartbeat Anti-Join: VMs with AMA Alive but No SecurityEvent (4h)

kql

let amaAlive = Heartbeat
    | where TimeGenerated > ago(1h)
    | where OSType == "Windows"
    | summarize LastHeartbeat = max(TimeGenerated) by Computer;
let eventSenders = SecurityEvent
    | where TimeGenerated > ago(4h)
    | summarize LastEvent = max(TimeGenerated) by Computer;
amaAlive
| join kind=leftanti eventSenders on Computer
| project Computer, LastHeartbeat
| sort by Computer asc

DCR Data Flow: Bytes and Events per DCR per Hour

kql

// Uses AzureMetrics if DCR metrics are routed to LAW (requires diagnostic settings on DCR)
AzureMetrics
| where TimeGenerated > ago(24h)
| where ResourceProvider == "MICROSOFT.INSIGHTS"
| where MetricName in ("LogsIngestionBytesPerMin", "LogsIngestionEventsPerMin")
| summarize AvgValue = avg(Average) by bin(TimeGenerated, 1h), Resource, MetricName
| order by TimeGenerated desc

Event Volume by EventID per VM (Last 24h)

kql

SecurityEvent
| where TimeGenerated > ago(24h)
| where EventID in (4624, 4625, 4648, 4688, 4698, 4720, 4728, 4732, 4756, 4768, 4769, 4776, 7045, 1102, 4103, 4104)
| summarize EventCount = count() by Computer, EventID
| sort by Computer asc, EventID asc

Detect Missing Required EventIDs (Compared to Expected Set)

kql

let requiredEvents = datatable(EventID:int, EventName:string, KillChainStage:string)
[
    4624,  "Logon Success",              "Lateral Movement",
    4625,  "Logon Failure",              "Initial Access",
    4648,  "Explicit Credential Logon",  "Lateral Movement",
    4688,  "Process Creation",           "Execution",
    4698,  "Scheduled Task Created",     "Persistence",
    4720,  "User Account Created",       "Persistence",
    4728,  "Member Added to Global Grp", "Privilege Escalation",
    4732,  "Member Added to Local Grp",  "Privilege Escalation",
    4756,  "Member Added to Univ Grp",   "Privilege Escalation",
    4768,  "Kerberos TGT Request",       "Credential Attack",
    4769,  "Kerberos TGS Request",       "Credential Attack",
    4776,  "NTLM Auth Attempt",          "Credential Attack",
    7045,  "New Service Installed",      "Persistence",
    1102,  "Security Log Cleared",       "Defense Evasion",
    4103,  "PowerShell Module Logging",  "Execution",
    4104,  "PowerShell Script Block",    "Execution"
];
let observedEvents = SecurityEvent
    | where TimeGenerated > ago(4h)
    | summarize EventCount = count() by EventID;
requiredEvents
| join kind=leftouter observedEvents on EventID
| extend EventCount = coalesce(EventCount, 0)
| extend Status = iff(EventCount == 0, "GAP - no events in 4h", "OK")
| project EventID, EventName, KillChainStage, EventCount, Status
| order by Status desc, EventID asc

4688 CommandLine Population Rate by VM

kql

SecurityEvent
| where TimeGenerated > ago(1h)
| where EventID == 4688
| summarize
    Total = count(),
    WithCommandLine = countif(isnotempty(CommandLine)),
    WithoutCommandLine = countif(isempty(CommandLine))
    by Computer
| extend CommandLinePct = round(100.0 * WithCommandLine / Total, 1)
| where WithoutCommandLine > 0
| order by WithoutCommandLine desc

VMs with WithoutCommandLine > 0 are missing the ProcessCreationIncludeCmdLine_Enabled registry key. The 4688 event is there but useless for detection without CommandLine.

Adding a New VM to Collection

Follow all three steps. Skipping any one step will result in a silent gap.

Step 1 — Install AMA Extension

Azure CLI — Windows:

bash

az vm extension set \
  --vm-name <vm-name> \
  --resource-group <rg> \
  --name AzureMonitorWindowsAgent \
  --publisher Microsoft.Azure.Monitor \
  --enable-auto-upgrade true

Azure CLI — Linux:

bash

az vm extension set \
  --vm-name <vm-name> \
  --resource-group <rg> \
  --name AzureMonitorLinuxAgent \
  --publisher Microsoft.Azure.Monitor \
  --enable-auto-upgrade true

Verify installation completed:

bash

az vm extension show \
  --vm-name <vm-name> \
  --resource-group <rg> \
  --name AzureMonitorWindowsAgent \
  --query "provisioningState" \
  --output tsv

Expected: Succeeded

If using Terraform, add the VM resource ID to var.monitored_vm_ids and use the azurerm_virtual_machine_extension resource:

hcl

resource "azurerm_virtual_machine_extension" "ama" {
  name                       = "AzureMonitorWindowsAgent"
  virtual_machine_id         = azurerm_windows_virtual_machine.vm.id
  publisher                  = "Microsoft.Azure.Monitor"
  type                       = "AzureMonitorWindowsAgent"
  type_handler_version       = "1.0"
  auto_upgrade_minor_version = true
  automatic_upgrade_enabled  = true
}

Step 2 — Associate the Kill Chain DCR

bash

az monitor data-collection rule association create \
  --name "assoc-kill-chain-<vm-name>" \
  --rule-id "/subscriptions/<sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.Insights/dataCollectionRules/cirius-kill-chain-dcr" \
  --resource "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Compute/virtualMachines/<vm-name>"

Or via Terraform — add the VM resource ID to the monitored_vm_ids variable and apply via PR.

Step 3 — Verify Data Flow

Allow 10–15 minutes after association. Then run:

kql

// Check Heartbeat and SecurityEvent for the new VM
Heartbeat
| where TimeGenerated > ago(1h)
| where Computer == "<vm-hostname>"
| summarize LastHeartbeat = max(TimeGenerated)

kql

SecurityEvent
| where TimeGenerated > ago(30m)
| where Computer == "<vm-hostname>"
| summarize EventCount = count() by EventID
| sort by EventID asc

If Heartbeat appears but SecurityEvent does not, the AMA is connected but the DCR association is not being picked up. Restart AMA on the VM:

powershell

Restart-Service -Name AzureMonitorAgent -Force

DCR for Linux VMs

Linux VMs use AMA to collect syslog. Auditd events are forwarded to syslog by audispd (the audit dispatcher) and then captured by AMA via the syslog data source.

Linux Syslog DCR — Terraform

hcl

resource "azurerm_monitor_data_collection_rule" "linux_syslog" {
  name                = "cirius-linux-syslog-dcr"
  resource_group_name = "rg-logging-logs"
  location            = "eastus"

  destinations {
    log_analytics {
      workspace_resource_id = azurerm_log_analytics_workspace.central.id
      name                  = "law-central"
    }
  }

  data_flow {
    streams      = ["Microsoft-Syslog"]
    destinations = ["law-central"]
  }

  data_sources {
    syslog {
      name           = "auditd-syslog"
      facility_names = ["auth", "authpriv", "daemon", "kern", "syslog", "user"]
      log_levels     = ["Debug", "Info", "Notice", "Warning", "Error", "Critical", "Alert", "Emergency"]
      streams        = ["Microsoft-Syslog"]
    }
  }

  tags = {
    Environment = "prod"
    Owner       = "infrastructure"
    Project     = "cirius-group"
    ManagedBy   = "terraform"
    Lifecycle   = "permanent"
  }
}

Events land in the Syslog table in cirius-logging-law-central.

Key Syslog Facilities for auditd

Facility	Contains
`auth`	Authentication events (su, sudo, login)
`authpriv`	Sensitive authentication (SSH private key auth)
`daemon`	System daemon events
`kern`	Kernel messages (auditd typically uses this or `user`)
`user`	User-level syslog messages

Required Linux Configuration

For auditd events to reach AMA via syslog:

auditd must be installed and running: systemctl status auditd
audispd (audit dispatcher) must be configured to forward to syslog. On modern systems with auditd >= 3.0, this is the syslog plugin in /etc/audit/plugins.d/syslog.conf — set active = yes.
The AMA syslog DCR must be associated to the Linux VM.
rsyslog or syslog-ng must be configured to listen on the local socket that audispd writes to.

Verify auditd is forwarding to syslog on the VM:

bash

sudo grep "active" /etc/audit/plugins.d/syslog.conf
# Expected: active = yes

KQL — confirm Linux auditd events are reaching LAW:

kql

Syslog
| where TimeGenerated > ago(1h)
| where Facility in ("auth", "authpriv", "kern", "user")
| where SyslogMessage contains "audit"
| summarize Count = count() by Computer, Facility
| sort by Count desc

Terraform Patterns: Complete Reference

Variable Definitions

hcl

variable "monitored_vm_ids" {
  description = "Resource IDs of all Azure VMs to associate with the kill chain DCR"
  type        = list(string)
  default     = []
}

variable "linux_vm_ids" {
  description = "Resource IDs of Linux VMs to associate with the syslog DCR"
  type        = list(string)
  default     = []
}

Data Reference — Existing LAW

If the LAW is managed in a different Terraform root module, use a data source:

hcl

data "azurerm_log_analytics_workspace" "central" {
  name                = "cirius-logging-law-central"
  resource_group_name = "rg-logging-logs"
}

Replace azurerm_log_analytics_workspace.central.id with data.azurerm_log_analytics_workspace.central.id in the DCR resource.

Complete Association Block

hcl

resource "azurerm_monitor_data_collection_rule_association" "kill_chain_windows" {
  for_each = toset(var.monitored_vm_ids)

  name                    = "assoc-kc-${substr(md5(each.value), 0, 8)}"
  target_resource_id      = each.value
  data_collection_rule_id = azurerm_monitor_data_collection_rule.kill_chain.id
  description             = "Kill chain DCR association"
}

resource "azurerm_monitor_data_collection_rule_association" "syslog_linux" {
  for_each = toset(var.linux_vm_ids)

  name                    = "assoc-syslog-${substr(md5(each.value), 0, 8)}"
  target_resource_id      = each.value
  data_collection_rule_id = azurerm_monitor_data_collection_rule.linux_syslog.id
  description             = "Syslog DCR association for Linux VM"
}

The md5 truncation on the name avoids the 44-character name limit on association resources when VM names are long. It is deterministic — same VM ID always produces the same name.

Routine Monthly Verification

Run this checklist the first Monday of each month. Target workspace: cirius-logging-law-central (ID: 5d76d1f2).

1 — All VMs Are Associated to the Kill Chain DCR

bash

# Get all VMs in Azure prod subscription
az vm list --query "[].{Name:name, RG:resourceGroup, ID:id}" --output table

# Get all VMs associated to the kill chain DCR
az monitor data-collection rule association list \
  --rule-name cirius-kill-chain-dcr \
  --resource-group rg-logging-logs \
  --query "[].targetResourceId" \
  --output tsv

Cross-reference the two lists. Any VM not in the DCR association list is a gap.

2 — No Data Gaps in the Last 30 Days

kql

// Check for any 4-hour windows with zero SecurityEvents from any VM
let amaDevices = Heartbeat
    | where TimeGenerated > ago(30d)
    | where OSType == "Windows"
    | summarize by Computer;
let secEventDevices = SecurityEvent
    | where TimeGenerated > ago(30d)
    | summarize LastEvent = max(TimeGenerated) by Computer;
amaDevices
| join kind=leftouter secEventDevices on Computer
| extend DaysSinceLastEvent = datetime_diff("day", now(), LastEvent)
| where DaysSinceLastEvent > 1 or isnull(LastEvent)
| project Computer, LastEvent, DaysSinceLastEvent
| sort by DaysSinceLastEvent desc

VMs with DaysSinceLastEvent > 1 or null LastEvent require investigation.

3 — Review Event Volumes for Anomalies

kql

// Daily event volume per EventID for the last 30 days — look for sudden drops
SecurityEvent
| where TimeGenerated > ago(30d)
| where EventID in (4624, 4625, 4648, 4688, 4698, 7045, 1102, 4103, 4104)
| summarize EventCount = count() by bin(TimeGenerated, 1d), EventID
| order by TimeGenerated desc, EventID asc

Flag any EventID that drops to zero for a day in the middle of the 30-day window. A sustained drop is likely a DCR or audit policy change that went unnoticed. A brief drop may indicate a maintenance window or agent restart.

4 — AMA Extension Version Check

bash

az vm extension list \
  --query "[?name=='AzureMonitorWindowsAgent' || name=='AzureMonitorLinuxAgent'].{VM:virtualMachine.id, Version:typeHandlerVersion, AutoUpgrade:enableAutomaticUpgrade}" \
  --output table

Confirm enableAutomaticUpgrade is true on all extensions. If any show false, auto-upgrade is disabled — extensions on those VMs will not receive security patches automatically.

5 — Verify No MMA Agent Is Present on Any VM

bash

az vm extension list \
  --query "[?name=='MicrosoftMonitoringAgent' || name=='OmsAgentForLinux'].{VM:virtualMachine.id, Name:name}" \
  --output table

Any result here indicates a VM still running the legacy MMA agent. It must be removed before AMA will collect reliably from that VM.

6 — Confirm Sentinel Is Still Enabled on law-central

bash

az security workspace-setting list --output table

cirius-logging-law-central must appear in this list. If it does not, Sentinel has been disabled and all Microsoft-SecurityEvent DCRs will fail on next modification.

Kill Chain Phase 1 Enablement — audit policy deployment (Intune and GPO) that generates the events DCRs collect
Kill Chain Audit Policy — narrower earlier reference
Kill Chain Incident Response
Log Retention Verification
Azure Monitor Alerts
KQL Query Reference
Terraform State Management

Document History

Date	Change	Author
2026-05-25	Initial — full DCR/AMA lifecycle: concepts, listing, Terraform patterns, AMA health checks, troubleshooting, KQL monitoring, Linux syslog DCR, monthly verification checklist	Rory / Kobe

DCR and AMA Management Runbook ​

Purpose ​

Critical Gotchas — Read Before Touching Anything ​

1. Microsoft-SecurityEvent Stream Requires a Sentinel-Enabled Workspace ​

2. Do Not Use Microsoft-Event as a Workaround ​

3. PowerShell 4103/4104 Come from a Different Log Channel ​

4. EventID 7045 Lives in the System Log, Not the Security Log ​

5. Never Mix AMA and MMA on the Same VM ​

Concepts ​

DCR Components ​

DCR Associations ​

Windows Event Log DCR vs Custom Log DCR ​

Listing and Inspecting DCRs ​

Azure CLI ​

Azure Portal ​

Creating and Updating DCRs ​

Terraform: Main Kill Chain DCR ​

XPath Filter Reference ​

Terraform: DCR Association to VMs ​

CLI: Manual DCR Association (Out-of-Band Fix) ​

Importing Existing DCRs into Terraform State ​

AMA Health Checks ​

Verify AMA Is Installed on a VM ​

Verify AMA Service Is Running ​

AMA Log Locations ​

Restart AMA (When Needed) ​

Troubleshooting — Data Not Flowing ​

Step 1 — Verify AMA Service Is Running on the VM ​

Step 2 — Check DCR Association Exists for That VM ​

Step 3 — Verify XPath Filter Syntax ​

Step 4 — Check AMA Logs for Upload Errors ​

Step 5 — KQL: VMs Sending Heartbeat but No SecurityEvent ​

Step 6 — Verify Workspace Is Sentinel-Enabled ​

Step 7 — Check DCR Data Flow Metrics in Azure Monitor ​

KQL Queries for Monitoring ​

Heartbeat Anti-Join: VMs with AMA Alive but No SecurityEvent (4h) ​

DCR Data Flow: Bytes and Events per DCR per Hour ​

Event Volume by EventID per VM (Last 24h) ​

Detect Missing Required EventIDs (Compared to Expected Set) ​

4688 CommandLine Population Rate by VM ​

Adding a New VM to Collection ​

Step 1 — Install AMA Extension ​

Step 2 — Associate the Kill Chain DCR ​

Step 3 — Verify Data Flow ​

DCR for Linux VMs ​

Linux Syslog DCR — Terraform ​

Key Syslog Facilities for auditd ​

Required Linux Configuration ​

Terraform Patterns: Complete Reference ​

Variable Definitions ​

Data Reference — Existing LAW ​

Complete Association Block ​

Routine Monthly Verification ​

1 — All VMs Are Associated to the Kill Chain DCR ​

2 — No Data Gaps in the Last 30 Days ​

3 — Review Event Volumes for Anomalies ​

4 — AMA Extension Version Check ​

5 — Verify No MMA Agent Is Present on Any VM ​

6 — Confirm Sentinel Is Still Enabled on law-central ​

Related Documentation ​

Document History ​

DCR and AMA Management Runbook

Purpose

Critical Gotchas — Read Before Touching Anything

1. Microsoft-SecurityEvent Stream Requires a Sentinel-Enabled Workspace

2. Do Not Use Microsoft-Event as a Workaround

3. PowerShell 4103/4104 Come from a Different Log Channel

4. EventID 7045 Lives in the System Log, Not the Security Log

5. Never Mix AMA and MMA on the Same VM

Concepts

DCR Components

DCR Associations

Windows Event Log DCR vs Custom Log DCR

Listing and Inspecting DCRs

Azure CLI

Azure Portal

Creating and Updating DCRs

Terraform: Main Kill Chain DCR

XPath Filter Reference

Terraform: DCR Association to VMs

CLI: Manual DCR Association (Out-of-Band Fix)

Importing Existing DCRs into Terraform State

AMA Health Checks

Verify AMA Is Installed on a VM

Verify AMA Service Is Running

AMA Log Locations

Restart AMA (When Needed)

Troubleshooting — Data Not Flowing

Step 1 — Verify AMA Service Is Running on the VM

Step 2 — Check DCR Association Exists for That VM

Step 3 — Verify XPath Filter Syntax

Step 4 — Check AMA Logs for Upload Errors

Step 5 — KQL: VMs Sending Heartbeat but No SecurityEvent

Step 6 — Verify Workspace Is Sentinel-Enabled

Step 7 — Check DCR Data Flow Metrics in Azure Monitor

KQL Queries for Monitoring

Heartbeat Anti-Join: VMs with AMA Alive but No SecurityEvent (4h)

DCR Data Flow: Bytes and Events per DCR per Hour

Event Volume by EventID per VM (Last 24h)

Detect Missing Required EventIDs (Compared to Expected Set)

4688 CommandLine Population Rate by VM

Adding a New VM to Collection

Step 1 — Install AMA Extension

Step 2 — Associate the Kill Chain DCR

Step 3 — Verify Data Flow

DCR for Linux VMs

Linux Syslog DCR — Terraform

Key Syslog Facilities for auditd

Required Linux Configuration

Terraform Patterns: Complete Reference

Variable Definitions

Data Reference — Existing LAW

Complete Association Block

Routine Monthly Verification

1 — All VMs Are Associated to the Kill Chain DCR

2 — No Data Gaps in the Last 30 Days

3 — Review Event Volumes for Anomalies

4 — AMA Extension Version Check

5 — Verify No MMA Agent Is Present on Any VM

6 — Confirm Sentinel Is Still Enabled on law-central

Related Documentation

Document History