Runbook: Incident Response

Purpose

This runbook provides step-by-step response procedures for common security and operational incidents. It is intended to be referenced during an active incident to ensure a consistent, complete response.

Note: This environment experienced a ransomware attack in November 2024. The procedures in this runbook incorporate lessons learned from that event.

Incident Severity Levels

Severity	Description	Response Time
P1 — Critical	Active breach, ransomware, complete outage	Immediate
P2 — High	Compromised account, significant service degradation	Within 1 hour
P3 — Medium	Single server down, suspicious activity detected	Within 4 hours
P4 — Low	Minor issues, informational alerts	Next business day

General Incident Steps

For any incident, always start here before moving to the specific procedure:

1. Document the time the incident was discovered
2. Assess severity using the table above
3. Notify leadership for P1 and P2 incidents immediately
4. Do not attempt to clean up or remediate before collecting evidence — preserve logs
5. Open an incident log and record all actions taken with timestamps

---

1. Ransomware / Security Breach

⚠️ P1 — Immediate response required

This is the highest severity incident. Do not attempt to negotiate with attackers or pay ransoms without explicit leadership direction.

Lessons from November 2024 incident incorporated below.

Detection Signals — What to Expect First

The November 2024 ransomware incident demonstrated that detection typically surfaces through multiple channels simultaneously. In a real event, expect:

• Cortex XDR — behavioral alerts on affected Windows endpoints (process injection, unusual LSASS access, lateral movement)
• SecOps platform — kill-chain agents will fire for execution/persistence/credential-dump/lateral-movement stages if EventID 4688, 7045, 4698 are flowing
• Arctic Wolf MDR — Arctic Wolf monitors all VLCs continuously and will alert independently; they may call before you see the alert
• User reports — typically the first human signal: "files are encrypted" or "I can't access network shares"
• Palo Alto DNS Security — may block C2 callback domains before encryption begins; check firewall threat logs

If only one signal is present, treat it as high-severity until disproved. Do not wait for multiple confirmations before beginning containment.

Communication Checklist

Complete this in parallel with technical containment — do not delay notification waiting for full investigation:

Person	When	Method	Purpose
Greg (T1 DR Admin)	Immediately	Phone	DR escalation, leadership awareness
Kevin (T1 DR Admin)	Immediately	Phone	DR escalation, leadership awareness
Arctic Wolf MDR	Within 5 minutes	Arctic Wolf portal → open P1 case	Activate MDR IR assistance; they have act-first authorization
Paul	Within 1 hour	Phone	FTP customer relationship owner — customer comms
Adriana	If ePHI breach suspected	Phone	BAA/vendor/risk docs, breach notification process
Legal counsel	If ePHI breach confirmed	Phone	Before sending any breach notifications
IR Retainer	If scope beyond internal capacity	See IR Retainer Engagement	External IR firm engagement

Immediate Actions (First 30 Minutes)

1. Isolate affected systems immediately

   • Disconnect affected VMs from the network: Azure portal → VM → Networking → disable NIC
   • For VMs not reachable via portal: Palo Alto → add block rule for affected VM private IP
   • Do NOT shut down affected systems — running memory may contain forensic evidence
   • Do NOT delete anything

2. Notify leadership and Arctic Wolf — see Communication Checklist above

3. Preserve evidence — before any remediation

   • Export Azure Activity Logs (portal → Monitor → Activity Log → export, last 90 days)
   • Export Entra ID sign-in logs (Entra ID → Monitoring → Sign-in logs)
   • Capture Palo Alto threat and traffic logs for the 24-hour window before discovery
   • Screenshot SecOps incident dashboard — capture current incident state
   • Export Cortex XDR alerts for affected endpoints
   • Note: S3 archive bucket cirius-archive-logs-prod contains syslog-ng archives if the syslog VM was not compromised

4. Assess blast radius

   • Which systems are confirmed affected? (Cortex XDR → Incidents for affected hostnames)
   • Is ePHI accessible on affected systems? (refer to CMDB PHI adjacency flags)
   • Has the attacker moved laterally? (LAW query: SignInLogs where AccountDisplayName contains compromised account, last 48h)
   • Have any backup jobs been deleted or disabled? (Veeam console → check job history)

5. Lock down Azure access — in this order

   1. Revoke all active sessions for confirmed compromised accounts: Entra ID → Users → [user] → Revoke sessions
   2. Disable confirmed compromised accounts: Entra ID → Users → [user] → Account enabled → Off
   3. Reset GlobalProtect VPN credentials for affected users
   4. Review and revoke suspicious OAuth app permissions: Entra ID → Enterprise Applications → filter by consent date
   5. Reset all privileged account passwords if breach scope is unclear

6. Azure subscription lockdown (if breach scope is broad)

   • Azure portal → Subscriptions → [subscription] → Resource locks → Add lock type Delete+ReadOnly
   • This prevents attackers from deleting resources or modifying infrastructure while you contain
   • Document the lock — remember to remove it after containment before any Terraform applies

Containment

7. Block attacker at the Palo Alto firewall

   • Identify attacker IP addresses from threat logs and Entra sign-in logs
   • Panorama → Policies → Security → add deny rules for identified external sources
   • Consider enabling Palo Alto Zone Protection temporary block-all-new-sessions if breach is active

8. Assess whether DR failover is needed

   • If Azure is compromised, do NOT connect AWS DR environment to any potentially compromised systems
   • Do NOT sync, replicate, or copy any data from potentially compromised Azure to AWS
   • If workloads need to continue, follow the DR Failover Procedure using clean AWS environment only

Recovery

9. Veeam restore decision tree

   Restore in this order (dependency-driven):

   1. CGIRDPAZP01 (domain controller) — all other VMs depend on AD authentication
   2. BIZARCAZP01 (primary business application server)
   3. All remaining VMs in dependency order

   Before restoring:

   • Identify the last clean Veeam backup timestamp (before first IOC detection)
   • Do NOT restore any backup taken after the suspected initial compromise date
   • Veeam nightly runs complete around 06:00 UTC — check backup job completion logs

10. Rebuild affected systems from clean backups

    • Use Veeam backups from before the compromise date
    • Restore to isolated network segment first — validate before connecting to main network
    • Verify backup integrity (Veeam → verify backup) before restoring

11. Validate restored systems before reconnecting to the network

    • Run Cortex XDR scan on restored VM before network reconnect
    • Verify no attacker artifacts remain (scheduled tasks, services, registry run keys)

12. Reset all credentials across the environment — assume all credentials are compromised

    • On-prem AD service accounts
    • Entra ID accounts and app registration secrets
    • Palo Alto local admin accounts
    • AWS root and IAM credentials if AWS environment was accessible

HIPAA Breach Notification

13. Assess whether ePHI was accessed or exfiltrated

    • If ePHI may have been compromised, HIPAA breach notification requirements apply
    • Engage legal counsel before sending any breach notifications
    • HIPAA requires notification to affected individuals within 60 days of discovering a breach
    • If breach affects 500+ individuals in a state: media notice also required
    • Business Associates must notify covered entity without unreasonable delay

14. Document everything — timeline, affected systems, data potentially exposed, actions taken

    Minimum documentation required for HIPAA breach assessment:

    • Date and time of initial compromise (estimated)
    • Date and time of discovery
    • Detection signal that surfaced it first
    • Nature of ePHI on affected systems
    • Number of individuals potentially affected
    • Actions taken and timestamps

---

2. Azure Outage / DR Failover

⚠️ P1 if affecting production — P2 if partial degradation

1. Confirm the outage — check Azure Service Health at status.azure.com

   • Is this a regional outage or isolated to specific services?
   • Is this affecting all Azure or specific resource types?

2. Assess impact

   • Which systems are affected?
   • Are customers impacted?
   • Is ePHI at risk?

3. Determine if DR failover is needed

   • Partial outage: wait for Azure recovery if ETA is acceptable
   • Full regional outage or extended downtime: initiate DR failover

4. If initiating DR failover follow the DR Failover Procedure completely

5. Notify leadership of status and decision to failover if applicable

6. Notify customers if their access is impacted

7. Monitor Azure recovery — when Azure is restored, plan failback

---

3. Server Down

⚠️ P2 if production — P3 if dev/non-critical

1. Confirm the server is actually down

   • Check Azure portal — is the VM running?
   • Check Palo Alto logs — is traffic reaching the server?
   • Attempt RDP connection from management zone

2. Check for obvious causes

   • Azure portal → VM → Activity Log — any recent changes?
   • Check if the VM was accidentally stopped or deallocated
   • Check disk space — full disk can cause services to fail

3. Attempt to start the VM if it is stopped

# Via Azure CLI if portal is unavailable
az vm start --resource-group <rg-name> --name <vm-name>

4. If the VM starts — verify services are running after boot

Get-Service | Where-Object {$_.Status -ne 'Running'}

5. If the VM will not start — check for platform issues in Azure portal, review boot diagnostics

6. If services are down but VM is running — restart the affected services per the Common Tasks runbook

7. If the VM is unrecoverable — restore from Veeam backup

   • Open Veeam console
   • Locate the most recent clean backup
   • Restore to Azure
   • Validate the restored VM

8. Document the cause and resolution

---

4. Data Breach Notification

⚠️ P1 — Immediate legal and leadership notification required

A data breach involves unauthorized access to or disclosure of ePHI. HIPAA has strict notification requirements.

Immediate Steps

1. Notify leadership immediately — do not delay

2. Preserve all evidence — do not modify or delete any logs or systems

3. Assess the breach scope

   • What data was accessed or disclosed?
   • How many individuals are affected?
   • What is the nature of the ePHI involved?
   • When did the breach occur vs. when was it discovered?

4. Contain the breach — stop any ongoing unauthorized access

HIPAA Notification Requirements

Notification	Requirement	Deadline
Affected individuals	Written notice required	Within 60 days of discovery
HHS (Health and Human Services)	Required for all breaches	Within 60 days of discovery
Media notice	Required if 500+ individuals in a state affected	Within 60 days of discovery
Business Associates	Must notify covered entity	Without unreasonable delay

5. Engage legal counsel before sending any breach notifications

6. Document the breach in full:

   • Date and time of breach
   • Date and time of discovery
   • Nature of ePHI involved
   • Number of individuals affected
   • Actions taken in response

7. Conduct root cause analysis and implement controls to prevent recurrence

---

5. Compromised User Account

⚠️ P2 — Respond within 1 hour

Signs of a compromised account include: login from unexpected location, login at unusual hours, MFA prompts the user did not initiate, suspicious activity in audit logs.

Immediate Containment

1. Disable the account immediately in Entra ID

   • Entra ID → Users → select user → Account → toggle Account enabled to Off

2. Revoke all active sessions

   • Entra ID → Users → select user → Revoke sessions

3. Notify the user through an out-of-band channel (phone, not email in case email is compromised)

4. Notify leadership if the account had access to ePHI

Investigation

5. Review Entra ID sign-in logs for the account

   • When did the unauthorized access occur?
   • What IP addresses were used?
   • What resources were accessed?

6. Review audit logs for actions taken by the compromised account

   • Were any changes made to infrastructure or data?
   • Were any other accounts modified or created?
   • Was any data exported or downloaded?

7. Check for persistence mechanisms

   • Were any new OAuth app permissions granted?
   • Were any new accounts created?
   • Were any firewall rules changed?

8. Assess ePHI exposure

   • Did the account have access to ePHI?
   • Is there evidence ePHI was accessed or exfiltrated?
   • If yes — follow the Data Breach Notification procedure

Recovery

9. Re-enable the account only after the compromise vector is identified and closed

10. Require password reset and MFA re-enrollment before restoring access

11. Review and confirm the account's access permissions are still appropriate — remove anything unnecessary

12. Block attacker IPs identified in the sign-in logs at the Palo Alto firewall

---

6. IR Retainer Engagement

⚠️ For P1 incidents that exceed internal response capacity

An IR retainer provides rapid access to forensic and incident response specialists when internal capacity is insufficient or when an incident requires third-party evidence collection (e.g. for legal hold or insurance claims).

Contact details: TBD — pending retainer selection and contract execution (IRRET-004/005). Update this section with firm name, 24/7 hotline, account number, and primary contact once IRRET-005 is complete.

When to Engage the Retainer

Trigger the retainer call if ANY of the following conditions are met:

• Ransomware encryption is confirmed or actively in progress
• ePHI is confirmed to have been exfiltrated
• An internal threat actor (employee or contractor) is suspected
• The scope of compromise spans more than 3 systems and is still expanding
• Arctic Wolf MDR has recommended external IR escalation
• Legal counsel has advised third-party forensic collection is needed for the claim
• Internal team capacity is exhausted and the incident is not contained after 4 hours

When in doubt: call. The cost of an unused retainer hour is far less than delayed containment.

Who Makes the Call

Primary: Rory — sole IT/security administrator, first responder for all incidents.

If Rory is unreachable: Greg or Kevin (T1 Domain Admins with break-glass access) should contact the retainer directly and immediately notify Rory.

Do not wait for leadership approval before calling. Authorization to engage is implicit for any P1 qualifying condition above.

Before You Call — Handoff Checklist

Have the following ready when you dial the retainer. The intake analyst will ask for all of it:

• [ ] Company name and primary contact name
• [ ] Incident type (ransomware / account compromise / data exfiltration / insider threat / other)
• [ ] Time of initial discovery
• [ ] Best estimate of time of first compromise
• [ ] Detection signals that surfaced it (Cortex XDR, Arctic Wolf, user report, etc.)
• [ ] Number of confirmed affected systems and hostnames
• [ ] Whether ePHI is involved (yes / suspected / unknown)
• [ ] Whether Azure subscription lock is in place
• [ ] Whether you have an open Arctic Wolf P1 case number
• [ ] Whether any credentials have already been reset (tell them exactly which)
• [ ] Whether any systems have been shut down (forensic evidence may be lost)

What the Retainer Firm Needs Access To

Have these ready to share — the IR team will request them within the first hour:

• Azure portal access (Guest/Reader role or coordinate via Microsoft support case)
• Entra ID sign-in logs export
• Palo Alto threat log export (24-hour window around compromise)
• Cortex XDR console access or alert export
• SecOps incident dashboard URL and current state screenshot
• Veeam console access (read-only)
• S3 archive bucket cirius-archive-logs-prod access (read-only, coordinate with IR team for IAM role)

After Engagement

• Add the firm's case number to the SecOps incident ticket
• Log all IR firm actions in your incident timeline (who, what, when)
• Do not independently remediate affected systems while IR firm is conducting forensics — wait for their go-ahead
• Keep leadership (Greg, Kevin) updated on IR firm findings every 2 hours

---

Post-Incident Actions

For all P1 and P2 incidents, complete the following after the incident is resolved:

• [ ] Incident timeline documented (start, discovery, containment, resolution)
• [ ] Root cause identified
• [ ] Contributing factors documented
• [ ] Controls implemented to prevent recurrence
• [ ] Lessons learned documented
• [ ] Leadership briefed on findings
• [ ] HIPAA breach assessment completed and documented
• [ ] Documentation updated if procedures changed

---

Document History

Date	Change	Author
April 2026	IRRET-006: Added Section 6 — IR Retainer Engagement: escalation triggers, who calls, pre-call handoff checklist, access requirements, post-engagement guidance. Contact details TBD pending IRRET-005.	Kobe
April 2026	DR-017: Added November 2024 lessons — detection signals section, communication checklist (Greg/Kevin/Arctic Wolf/Paul/Adriana), structured containment with Azure subscription lockdown, Veeam restore decision tree, HIPAA breach documentation requirements	Kobe
February 2026	Initial draft	Rory

Related Documentation

• DR Failover Procedure
• Common Tasks Runbook
• HIPAA Controls Matrix
• Vulnerability Management
• Palo Alto Firewall Overview