Runbook: Disaster Recovery Test Checklist

Purpose

A quarterly disaster recovery test validates that backups restore, replicas boot, and recovery time objectives are met before an actual incident forces them to. This checklist is the authoritative DR test procedure — any step skipped must be documented and compensating validation performed.

Estimated time: 4–6 hours. Run quarterly (first week of each quarter). Publish results to SharePoint compliance folder.

---

0. Pre-Test

• [ ] Notify Rory, Kevin, Greg that DR test is starting (Teams + email)
• [ ] Confirm production is stable and no active incidents are open
• [ ] Open change ticket in SecOps: POST /api/changes with summary "Quarterly DR test — Q<N> <YYYY>"
• [ ] Snapshot current RPO/RTO targets from SOC2 evidence folder for comparison
• [ ] Create a dedicated SharePoint folder for this test: SharePoint → Compliance → DR Tests → <YYYY>-Q<N>

---

1. Veeam Replica Boot Test

Veeam replicates critical VMs from Azure PROD to AWS Backup account via the Veeam proxy in the Backup account. Boot a replica in an isolated network to verify the VM starts, the OS loads, and key services respond.

• [ ] Log into Veeam console (jump via Twingate → Veeam management VM)
• [ ] Select replica for test boot — rotate VMs across quarters (target 4 VMs/year minimum)
  • Q1: Domain controller
  • Q2: SQL database server
  • Q3: Application server (Medicare workflow host)
  • Q4: File server
• [ ] Right-click replica → Failover Plan → Test Failover
• [ ] Verify VM boots into the isolated DR test VNet (no connectivity to prod)
• [ ] RDP or console into the booted VM
• [ ] Verify OS boot completed cleanly — no filesystem errors, no missing drivers
• [ ] Verify key services started (AD DS, SQL, IIS, or app-specific)
• [ ] Capture screenshot of services console showing RUNNING state
• [ ] Record boot duration (power-on → login prompt)
• [ ] Undo Failover when validation complete — replica returns to replication state

Pass criteria: VM boots within 15 minutes, OS and key services healthy, no data corruption visible.

---

2. Recovery Services Vault Restore Test

Azure Recovery Services Vaults (RSV) hold Azure Backup snapshots with soft delete + 7 year immutability. Test a file-level restore and a full VM restore to validate both recovery paths.

2a. File-level restore

• [ ] Select a recent recovery point from RSV rsv-prod-backup-central
• [ ] Restore Files workflow → mount iSCSI target to recovery VM
• [ ] Copy 3 sample files from different paths (confirms volume mount works)
• [ ] Verify file contents match a known-good hash (store baseline in SharePoint)
• [ ] Unmount iSCSI target cleanly

2b. Full VM restore

• [ ] Select a test VM recovery point (non-production workload — use the DR test VM)
• [ ] Restore to a new VM in the DR test resource group rg-dr-test
• [ ] Verify restored VM powers on and OS boots
• [ ] Verify data disk is attached and accessible
• [ ] Delete restored VM and resource group when validation complete

Pass criteria: File restore completes in under 30 minutes, full VM restore completes in under 2 hours, restored data passes hash check.

---

3. Smoke Tests

Verify that fundamental connectivity and identity work in a DR scenario.

• [ ] From the test-failover domain controller, run dcdiag /v — all tests pass
• [ ] Test AD authentication against the DR DC using a test service account
• [ ] Verify DNS resolution for critical internal hostnames
• [ ] Verify time sync (w32tm /query /status) — NTP must work for Kerberos
• [ ] Launch Medicare workflow app on DR app server, complete a dry-run transaction
• [ ] Verify SQL replica accepts connections and returns test query results
• [ ] Check Palo Alto DR firewall health — Panorama shows Connected

Pass criteria: All smoke tests pass. Document any warnings, even non-blocking ones.

---

4. RTO Measurement

Record the actual time required for each recovery operation to compare against target RTOs in the SOC2 evidence folder.

Operation	Target RTO	Actual	Notes
Veeam replica boot (single VM)	15 min
RSV file-level restore	30 min
RSV full VM restore	2 hr
AD authentication from DR DC	30 min
Medicare workflow app functional	4 hr
Full DR site operational (all services)	8 hr

Pass criteria: All operations meet target RTO. Any miss triggers a remediation ticket.

---

5. Velociraptor Verification

Velociraptor is the incident response triage tool — we must confirm it is operational before a real incident forces us to rely on it. A DR scenario is also a likely trigger for IR activity, so Velociraptor needs to work in both the production and DR configurations.

5a. Velociraptor server health

• [ ] Velociraptor server is reachable via the admin UI over Twingate
• [ ] Server certificate is valid (not within 30 days of expiry)
• [ ] Server shows last 7 days of client check-ins in the Clients view
• [ ] Server disk usage is below 80% — purge old hunts if above

5b. Agent deployment coverage

• [ ] Count of online clients matches the expected managed endpoint count (±2 for offline laptops)
• [ ] Any endpoint offline for >14 days is investigated — either reinstall agent or decommission from CMDB
• [ ] Agents on all three tiers present: workstations, Windows servers, Linux VMs (Velociraptor supports Linux)

5c. Artifact collection dry-run

• [ ] Launch a lightweight hunt on 3 representative endpoints (one workstation, one server, one DC) using the Generic.Client.Info artifact
• [ ] Verify hunt completes within 10 minutes
• [ ] Download hunt results and confirm JSON is well-formed
• [ ] Delete hunt results after verification

5d. Velociraptor in the DR scenario

• [ ] Confirm Velociraptor server is not in the same failure domain as its clients — ideally deployed to DR region or AWS
• [ ] If clients come up in the DR environment (Veeam replica boots), verify agents can still check in to the Velociraptor server
• [ ] Document which artifacts would be run first during incident triage after a DR event (e.g. Windows.EventLogs.ServiceCreation, Windows.Registry.Sysinternals, Windows.Detection.Amcache)

Pass criteria: Server healthy, agent coverage matches expected count, hunt runs complete cleanly, DR communication path validated.

---

6. Post-Test

• [ ] Undo all test failovers — replicas returned to replication state
• [ ] Tear down any VMs created in rg-dr-test
• [ ] Confirm production has not been affected (spot-check monitoring)
• [ ] Close the SecOps change ticket with actual duration and outcome
• [ ] Publish test report to SharePoint: SharePoint → Compliance → DR Tests → <YYYY>-Q<N> → dr-test-report.pdf
• [ ] Include in report: RTO table, screenshots, pass/fail per section, remediation items
• [ ] File any remediation items as stories in SecOps project (DRSOP or equivalent)
• [ ] Notify Rory, Kevin, Greg of completion

---

Appendix A — Remediation History

Track failures and remediation across quarters so recurring issues are visible.

Quarter	Section	Issue	Remediation	Story ID

---

Appendix B — RTO/RPO Targets Source of Truth

The authoritative RTO/RPO targets live in:

• compliance/annual-review-checklist.md — business continuity section
• SharePoint → Compliance → SOC2 Evidence → Availability section

Update both this checklist and the SOC2 evidence binder if targets change.