Appearance
DR Failover Procedure
Purpose
This document provides step-by-step instructions for failing over from the primary Azure environment to the AWS disaster recovery environment. This procedure should be followed when the Azure environment is unavailable or compromised and business operations need to continue from AWS.
Recovery Objectives
| Objective | System | Target | Current State |
|---|---|---|---|
| RTO | CGIRDPAZP01, BIZARCAZP01, psql-secops-prod | 4 hours | Untested — Q2 2026 DR test will measure |
| RTO | All other systems | 8 hours | Untested — Q2 2026 DR test will measure |
| RPO | All systems | 1 hour (aspirational) | ~24 hours — Veeam nightly replication to S3 |
The RPO of approximately 24 hours reflects the nightly Veeam backup cycle. The 1-hour RPO target requires hourly or near-continuous Veeam replication — a planned 2026 improvement. Data created or modified since the last backup will need to be recovered manually or re-entered by customers.
See DR Overview — RTO/RPO Targets for formal targets, tracking table, and Greg's sign-off status.
Prerequisites
Before starting failover confirm the following:
- [ ] AWS console access is available
- [ ] AWS credentials and access verified
- [ ] Veeam backup console access is available
- [ ] S3 bucket containing latest Veeam backup template is accessible
- [ ] Contact list for leadership and customers is available
Stakeholder Notification
Notify the following parties before or immediately after declaring a DR event:
| Stakeholder | When to Notify | Method |
|---|---|---|
| Leadership / Management (Greg, Kevin) | Immediately upon declaring DR event | Phone / email |
| IT (Rory) | Immediately — leads the failover | N/A — initiating |
| Arctic Wolf MDR | Immediately — notify of DR event so MDR team is alert for anomalous activity | Arctic Wolf portal or phone |
| Paul | After environment is up — owns FTP customer relationships | Email / phone |
| Customers | After AWS environment is ready | Email / phone |
Draft customer notification template:
We are currently experiencing a service disruption in our primary environment. We have activated our disaster recovery environment and will notify you shortly with updated connection details. Please hold any pending data submissions until further notice.
Failover Steps
Phase 1 — Declare DR Event and Assess
Step 1: Confirm Azure is unavailable
Verify the Azure environment is genuinely unavailable or compromised before initiating failover. Check:
- Azure portal status at status.azure.com
- Whether the issue is regional (full failover needed) or isolated (may not require DR)
- Whether this is a security incident (ransomware, breach) — if so, do not connect AWS to any potentially compromised systems
Step 2: Declare DR event
Make the decision to failover and notify leadership immediately. Document the time the DR event was declared.
DR Event Declared: [DATE / TIME]
Reason: [DESCRIPTION]
Declared by: [NAME]Phase 2 — Bring Up AWS Infrastructure
Step 3: Log into AWS console
Log into the AWS management console using the Cirius Group management account (206820231356). Then switch to the appropriate account:
| Task | Account | Account ID |
|---|---|---|
| Start DR workload servers | Production | 807267566999 |
| Verify firewall | Networking | 238342914131 |
| Check backups | Backup | 863609217450 |
| Verify identity services | Identity | 414134953818 |
Use cross-account role switching — do not log into each account separately.
Step 4: Start EC2 instances
Start the EC2 instances in the AWS DR environment. Instances are pre-provisioned and ready — they just need to be started.
Priority start order:
- CGIRDPAZP01 (domain controller replica — must be up before other servers can authenticate)
- BIZARCAZP01 (business application server)
- All remaining DR instances
- Navigate to EC2 → Instances in the Production account (807267566999)
- Select DR instances in priority order above
- Actions → Start instances
- Wait for all instances to reach running state and pass both status checks (System and Instance)
Note: Twingate must be active to connect to EC2 instances in the private subnet — ensure Twingate access to the AWS DR environment is confirmed before attempting RDP/SSH.
Step 5: Verify network connectivity
Confirm the VPC, subnets, and Palo Alto firewall are operational:
- Verify Palo Alto VM-Series firewall is running in the Networking account (238342914131) and accessible via Panorama
- Confirm Panorama (PAN-OS 12.1.2) shows AWS DR firewall connected and policies pushed
- Confirm Transit Gateway and spoke VPC routes are active
- Verify instances can reach the internet and S3 via expected routes
- Verify Twingate connector in AWS DR is active and healthy
Phase 3 — Restore Database
Step 6: Locate latest Veeam backup in S3
Veeam sends a nightly server template from Azure to the designated S3 bucket. Locate the most recent backup:
- Navigate to S3 in the Prod account (807267566999) — bucket provisioned by
aws-infra/prod/veeam_s3.tf - Bucket:
cirius-veeam-prod-bucket - Confirm the date/time of the most recent Veeam template (sorted by LastModified)
- Note the RPO gap — any data after this backup timestamp will need to be recovered manually
- Validated Veeam targets: CGIRDPAZP01 and BIZARCAZP01 (validated March 4, 2026)
Step 7: Attach Veeam template to AWS database server
Restore the Veeam backup template to the AWS database server:
- Open Veeam backup console
- Locate the latest backup in the S3 repository
- Restore / attach the backup image to the designated AWS database EC2 instance
- Start the database service and verify it is operational
Step 8: Validate database
Confirm the database is running and accessible:
- [ ] Database service is running
- [ ] Database is accessible from application servers
- [ ] Spot check recent data to confirm backup integrity
- [ ] Document the backup timestamp — this is the RPO cut point
Phase 4 — Redirect Customers
Step 9: Update customer FTP endpoint
Customers currently send data to the Azure FTP endpoint. Notify each customer of the new AWS FTP endpoint and request they redirect their submissions:
- Compile the customer contact list
- Send individual notifications with the new AWS FTP endpoint details
- Request confirmation from each customer that they have updated their connection
Draft customer FTP redirect notification:
Our DR environment is now active. Please update your FTP connection to the following endpoint: [AWS FTP ENDPOINT]. Please resume data submissions at your earliest convenience and confirm receipt of this message.
Step 10: Update DNS / IP references
Update any internal DNS entries or hardcoded IP references to point to AWS:
- [ ] Internal DNS records updated to AWS IPs
- [ ] Application configuration files updated where needed
- [ ] Test connectivity from client systems to AWS endpoints
Phase 5 — Validate and Declare Operational
Step 11: Validate critical systems
Confirm all critical systems are operational in AWS before declaring the environment live:
- [ ] Database server running and accessible
- [ ] Customer FTP endpoint operational
- [ ] Palo Alto firewall enforcing security policies
- [ ] Basic connectivity tests passing from expected client sources
Step 12: Declare DR environment operational
Notify leadership that the DR environment is operational. Document the time.
DR Environment Operational: [DATE / TIME]
RTO Achieved: [TIME FROM DECLARATION TO OPERATIONAL]
RPO Cut Point: [DATE / TIME OF LAST VEEAM BACKUP]
Issues Encountered: [ANY NOTES]Known Gaps and Risks
| Gap | Risk | Mitigation |
|---|---|---|
| ~24 hour RPO | Up to 24 hours of data loss | Customers manually resend data after failover where possible |
| No automated data replication | Manual restore process increases RTO | Veeam nightly backup is current mitigation |
| Customer FTP redirect is manual | Delays in customer data submission | Maintain current customer contact list |
| Cortex XDR FQDN resolution in AWS | cortex-xdr.paloaltonetworks.com does not resolve in AWS — Cortex XDR traffic hits Containment deny rule | Apply VPC resolver 10.99.0.2 via Panorama device template (pending) |
| DR not yet tested | Unknown gaps in procedure | June 2026 DR test will validate this procedure |
Failback
Once the Azure environment is restored and stable, a separate failback procedure should be followed to return operations to Azure. See DR Failback Procedure for the return-to-Azure process (placeholder pending June 2026 DR test results).
Document History
| Date | Change | Author |
|---|---|---|
| April 2026 | DR-016: Updated RTO/RPO table with formal targets; named Veeam S3 bucket; added Arctic Wolf and Paul to stakeholder list; added EC2 start priority order and Twingate note; added Panorama verification to Step 5; added Cortex XDR FQDN gap to Known Gaps | Kobe |
| March 2, 2026 | Removed "coming soon" label from failback procedure link, replaced with placeholder context | Rory |
| March 2026 | Added AWS account switching details for Step 3 | Rory |
| February 2026 | Initial draft | Rory |