Appearance
Q2 2026 Disaster Recovery Test Plan
Purpose: Validate disaster recovery capability across Azure PROD, AWS DR, and the SecOps platform itself. Produce auditable evidence for HIPAA §164.308(a)(7)(ii)(D) — contingency plan testing and revision.
Owner: Rory (Security Officer) Test window: NOT COMPLETED — Q2 window missed (June 27 fallback passed). Reschedule to Q3 — must complete before September SOC2 audit. Proposed Q3 window: Saturday July 19, 2026. Rory to confirm with Kevin/Greg. Participants: Rory (lead), Kevin (T1 DR Admin), Greg (T1 DR Admin), Adriana (comms observer) Audience: Security Officer, auditors, leadership
Objectives
- Confirm RTO (Recovery Time Objective) for each critical service is under 4 hours
- Confirm RPO (Recovery Point Objective) for each critical service is under 1 hour
- Validate that Veeam replicas boot cleanly in the DR environment
- Validate that Recovery Services Vault restore path is independent and functional
- Validate that Twingate connectivity to DR Cloud PC works without PROD availability
- Validate that the SecOps platform (ca-secops-prod, psql-secops-prod) is recoverable
- Identify gaps for remediation before Q4 test
Scenarios
Scenario 1 — Full PROD failover simulation
Premise: Azure PROD tenant (ciriusgroup.com, subscription PROD) is assumed compromised by ransomware. All VMs untrusted. Rollback to last known clean state is not possible. Cut over to DR workloads in AWS Prod account (807267566999).
Activities:
- Declare disaster (Rory issues formal declaration in SecOps)
- Rotate all privileged credentials (Entra PROD break-glass only — not DDE)
- Spin up AWS DR Cloud PC via Twingate access
- Activate DR Ubuntu EC2 in private subnet (see
aws/dr-overview.md) - Bring up read-only SecOps UI pointing at last PITR restore of psql-secops-prod
- Validate user access via Twingate only (no GlobalProtect for DR path)
Scenario 2 — Veeam replica boot
Premise: Primary VMs unavailable; boot the most recent Veeam replica for three tier-1 workloads.
Targets:
- ACTDIRAZP01 (primary DC PROD)
- VM hosting SharePoint tenant backup connector
- Panorama management VM (boot from config backup if replica unavailable)
Validation:
- VM boots to login prompt
- AD replication resumes within 10 minutes or isolated forest mode functions
- SharePoint connector re-establishes session
- Panorama accepts a known-good push
Scenario 3 — Recovery Services Vault restore
Premise: Veeam is unavailable (ransomware impacted backup server). Restore must happen from Azure Recovery Services Vault only.
Targets:
- Restore one non-production VM to a new resource group
- Restore a single file-share item to DR RG
- Verify WORM lifecycle policy did not block retrieval
Validation:
- Restore completes within documented window
- File hash matches pre-backup hash
- Immutability policy (Unlocked) did not block legitimate restore
Scenario 4 — Twingate connectivity in DR mode
Premise: PROD tenant offline — Twingate must continue to function for DR administrators.
Validation:
- Twingate connectors in DDE and AWS networks remain "Online"
- Admin user can connect, reach DR Cloud PC, and reach AWS EC2 via SSH
- No dependency on PROD-only services (Entra PROD break-glass is the only PROD-hosted piece, verified independent)
Scenario 5 — SecOps platform availability
Premise: SecOps app in rg-logging-logs (logging subscription) must be available for incident coordination during the disaster.
Validation:
- SecOps UI reachable (read-only mode acceptable)
- API key authentication still works for critical POST operations
- PITR restore of psql-secops-prod completes within RTO
- Agent heartbeat shows findings ingestion resumes within RTO
Success criteria
| Metric | Target | How measured |
|---|---|---|
| RTO — primary DC | ≤ 4 hours | Time from declaration to successful domain login on DR |
| RTO — SecOps platform | ≤ 4 hours | Time from declaration to UI reachable in read-only mode |
| RTO — Twingate admin access | ≤ 30 minutes | Time from declaration to first successful admin session |
| RPO — psql-secops-prod | ≤ 1 hour | Delta between last committed row and restored row |
| RPO — SharePoint PHI | ≤ 1 hour | M365 retention policy verification |
| RPO — Veeam replicas | ≤ 15 minutes | Replica refresh interval (continuous data protection) |
| Critical apps accessible | 100% | DR admin can reach AD, SecOps, SharePoint, FTP customer portal |
| SecOps platform availability | Available read-only | POST/GET endpoints respond with expected codes |
Runbook
Pre-test (T-7 days)
- [ ] Confirm window with Kevin, Greg, Adriana (calendar invites)
- [ ] Send 7-day heads-up to SecOps subscribers — this will generate noise in incidents
- [ ] Open a maintenance window in SecOps (
POST /api/maintenance/active) covering test window - [ ] Snapshot psql-secops-prod immediately before test
- [ ] Confirm Cortex XDR weekly report shows no active HIGH/CRITICAL from past 7 days
- [ ] Rory prints (yes, paper) the runbook so it's available if laptops are out of scope
Pre-test (T-1 day)
- [ ] Verify DR Cloud PC EC2 boots cleanly from test image
- [ ] Verify Twingate connectors in AWS show "Online"
- [ ] Verify Recovery Services Vault has a restore point from the past 24 hours
- [ ] Verify Veeam job success for all protected VMs (past 24 hours)
Execute (T+0 to T+12h)
- [ ] 08:00 — Rory issues formal disaster declaration in SecOps
- [ ] 08:05 — Scenario 1 kickoff; Kevin spins up DR Cloud PC
- [ ] 08:30 — Scenario 2 kickoff; Greg boots first Veeam replica (ACTDIRAZP01)
- [ ] 09:00 — Scenario 3 kickoff; RSV test restore to new RG
- [ ] 09:30 — Scenario 4 kickoff; Twingate admin access validated
- [ ] 10:00 — Scenario 5 kickoff; SecOps PITR restore + read-only UI validation
- [ ] 12:00 — Checkpoint: all RTO targets measured; record in findings log
Validate (T+4h to T+8h)
- [ ] Each scenario lead writes a short outcome note with timestamps
- [ ] Adriana observes communication flow; notes any missing handoff
Failback (T+8h to T+11h)
- [ ] Decommission DR Cloud PC and test restores (teardown from SecOps via tagged-for-teardown)
- [ ] Close the maintenance window in SecOps
- [ ] Reconcile any incidents generated during the test (close as test-related)
Post-mortem (T+7 days)
- [ ] Rory convenes 90-minute post-mortem with Kevin, Greg, Adriana
- [ ] Write up findings in
compliance/q2-dr-test-findings-2026.md - [ ] Each gap becomes a SecOps story with owner and target
- [ ] Runbook updates committed as a follow-up PR
- [ ] Q4 DR test plan updated with Q2 lessons (see
q4-dr-test-plan-2026.md)
Out of scope for Q2
- Full rebuild of the Palo Alto NVA from backup (planned for Q4)
- DDE tenant cutover (handled separately — DDE is customer-facing, tested by DDE DR plan)
- Full data-plane failover for the Medicare published app
Known risks
| Risk | Mitigation |
|---|---|
| Test generates false-positive incidents in SecOps | Maintenance window suppresses alert noise |
| Veeam replica boot produces duplicate AD objects | Boot into isolated network only — no replication to production DCs |
| RSV restore consumes significant egress | Restore to DR RG, tagged for teardown, deleted within 48h |
| Failback misses a resource | Use tagged-for-teardown naming; quarterly unmanaged resource scan catches leftovers |
Document history
| Date | Change | Author |
|---|---|---|
| April 2026 | Initial Q2 DR test plan | Kobe |