VM Status Agent

Purpose: Operations guide for the VM status monitoring agent that checks VM health across all three environments (PROD Azure, DDE Azure, AWS) every 6 hours.

Audience: Rory Agent file: agents/operational/vm_status_agent.py

---

What It Checks

The VM status agent runs every 6 hours as part of the Container Apps Job. It queries VM power state across all environments and raises findings for unexpected state changes.

Environment	Source	How Queried
Azure PROD	Azure VM API (ciriusgroup.com subscription)	az vm list --show-details --subscription <prod-id>
Azure DDE	Azure VM API (ciriusdde.com subscription)	az vm list --show-details --subscription <dde-id>
AWS (all 7 accounts)	EC2 describe_instances per account	Boto3 via account-assumed role

---

Expected VM States

A VM's expected state is defined in the agent's config file: agents/operational/vm_status_config.json.

Always-running VMs (unexpected if stopped):

Environment	VMs	Reason
Azure PROD	Domain controllers, Panorama, Cortex XDR syslog VM	Core infrastructure
Azure DDE	DDE terminal servers, DDE domain controllers	Medicare app
AWS Networking (238342914131)	Palo Alto VM-Series firewall VMs	Network perimeter
AWS Logging (038901680748)	Syslog VM	Log collection
AWS Prod (807267566999)	Cloud PC (Ubuntu EC2), any DR workloads in active use	Cloud PC = Kobe's machine

Variable-state VMs (expected to start/stop):

• Any VM tagged Lifecycle=decommission — expected to be stopped or in the process of being deleted
• Dev environment VMs in AWS Dev account (040067931468) — expected to stop outside business hours

---

Severity

Condition	Severity
Always-running VM is stopped/deallocated	HIGH
Always-running VM stopped with no CM ticket in the last 2 hours	HIGH
Always-running VM stopped with a valid CM ticket	Suppressed (logged, no finding)
Dev VM running on weekend (04:00–20:00 PST Sat/Sun) without CM	LOW (cost alert)
VM in unknown state (transitioning)	LOW (informational)

A VM that should be running being stopped is potentially serious — an attacker can stop a VM to prevent forensic collection, to disrupt security tooling, or as part of a ransomware attack before encryption.

---

ioc_key Format

{source_system}:host:{vm_name}:unexpected_powerstate_stopped
{source_system}:host:{vm_name}:unexpected_powerstate_running

---

Responding to VM Status Findings

HIGH — always-running VM stopped:

1. Open SecOps finding — note source_system, VM name, timestamp
2. Check for a CM ticket: SecOps → Change Management → filter by host and time window
3. Check Azure Portal or AWS Console for current state and who stopped it:
   • Azure: VM blade → Activity Log → filter for "Stop", "Deallocate" events
   • AWS: CloudTrail → Event history → filter for StopInstances event → check userIdentity
4. If stopped by a legitimate admin with a CM ticket: create a known-good rule with expiry matching the CM window, close the finding
5. If stopped with no CM ticket and no explanation:
   • Is this a domain controller or security tool VM? → CRITICAL — escalate to Rory immediately, check for kill chain findings on the same or adjacent hosts
   • Is this a Palo Alto firewall VM? → CRITICAL — network perimeter may be down
   • Is this a less critical VM? → Contact the VM owner, restart if safe, escalate to Rory
6. Check all other kill chain findings in the same SecOps incident window — an unexpected VM shutdown during an active attack is a serious compound signal

LOW — dev VM running over weekend:

1. Identify who owns the VM (check Intune/CMDB Name tag)
2. Check if there is active work (SSH session, running jobs)
3. If idle: stop the VM to save cost — az vm stop or AWS Console stop
4. Remind the owner to stop dev VMs when done

---

Updating the Config

The list of always-running VMs lives in agents/operational/vm_status_config.json. Update this file when:

• A new permanent VM is deployed (add it to always-running)
• A VM is tagged for decommission (move it to decommission category)
• A VM name changes (Azure VM names are immutable — if the VM is recreated with a new name, update config)

{
  "always_running": {
    "PROD": ["vm-dc-prod-01", "vm-dc-prod-02", "panorama-prod"],
    "DDE": ["vm-dc-dde-01", "vm-ts-dde-01"],
    "AWS": ["i-xxxxxxxxxxxx"]
  },
  "decommission_bound": ["vm-oldserver-01"],
  "dev_accounts": ["040067931468"]
}

To update: edit the file, open a PR against bedrock-hub, merge. No restart required — the agent reads config on each run.

---

Monitoring the Agent

Check agent health in Container Apps Job logs:

ContainerAppConsoleLogs
| where TimeGenerated > ago(6h)
| where ContainerName contains "vm_status"
| project TimeGenerated, Log
| order by TimeGenerated desc

The agent logs a summary at the end of each run: how many VMs were checked per environment, how many findings were raised, and any API errors.

If the agent logs confidence: 0 in its result dict, it failed to query one or more environments. Check:

• Azure managed identity permissions (VM Reader on the relevant subscriptions)
• AWS assumed-role policy (EC2 DescribeInstances on all accounts)
• SecOps API availability

---

Related Documents

• runbooks/azure-daily-operations.md — Azure VM operations
• runbooks/cloud-pc-operations.md — Cloud PC (Ubuntu EC2) operations
• architecture/security-monitoring-architecture.md — full agent architecture
• runbooks/secops-change-management.md — CM ticket creation