Appearance
Backup Architecture
Purpose
This document describes the backup and disaster recovery replication architecture across all environments. It covers the primary Azure Recovery Services Vault backup solution, Veeam DR replication to AWS, vault hardening controls, and the backup verification process.
Architecture Overview
┌─────────────────────────────────────────────────────────┐
│ AZURE (Primary) │
│ │
│ VMs, SQL, File Shares │
│ │ │ │
│ ▼ ▼ │
│ Recovery Services Veeam Backup │
│ Vault (per-sub) Agent/Server │
│ (primary backup) │ │
└─────────────────────────────────────────────────────────┘
│
Nightly replication
(Veeam to AWS S3)
│
┌─────────────────────────────────────────────────────────┐
│ AWS (DR) │
│ │
│ Backup Account (863609217450) │
│ └── S3 Bucket (KMS encrypted, Object Lock, versioned) │
│ └── Veeam backup templates │
│ └── S3 lifecycle: Standard-IA 90d → Glacier 365d │
│ 6-year noncurrent expiration │
└─────────────────────────────────────────────────────────┘Two-Layer Backup Strategy
| Layer | Tool | Purpose | RPO | RTO |
|---|---|---|---|---|
| Primary backup | Azure Recovery Services Vault | VM restore, SQL database recovery, file-level recovery | 24 hours (daily) | 1–4 hours |
| DR replication | Veeam → AWS S3 | Full environment failover to AWS | ~24 hours (nightly) | 4–8 hours |
These are independent — Recovery Services Vault handles day-to-day backup and restore operations within Azure. Veeam handles the cross-cloud DR replication that enables failover to AWS.
Azure Recovery Services Vault (Primary Backup)
Production Tenant (azure-infra)
Recovery Services Vaults are deployed per-subscription via the recovery-vault module. Each vault has soft delete and immutability (Unlocked) enabled as defaults — a direct lesson from the November 2024 ransomware incident.
| Subscription | Vault | Schedule | Instant Retention | Long-term Retention |
|---|---|---|---|---|
| Production | rsv-prod-backup | Daily 2:00 AM UTC | 30 days | 12 weeks weekly, 12 months monthly |
| Identity | rsv-identity-backup | Daily 2:00 AM UTC | 30 days | 52 weeks weekly |
| Firewall | rsv-firewall-backup | Daily 3:00 AM UTC | 30 days | — |
| Dev | rsv-dev-backup | Daily 4:00 AM UTC | 14 days | — |
Protected resource types: Azure VMs (OS and data disks), Azure SQL databases, Azure File Shares where applicable.
Vault hardening — all active vaults:
- Soft delete: ✅ Enabled (14-day recovery window for deleted backup data)
- Immutability: ✅ Unlocked (prevents policy changes; can be escalated to Locked if needed)
- Minimum retention: ✅ 30-day daily verified on all active policies
Note on decommission-bound vaults: Vaults tagged Lifecycle = decommission intentionally do NOT have immutability enabled — immutability would block cleanup at the June 2026 destruction target.
DDE Tenant (azure-dde-infra)
Both DDE subscriptions now have Recovery Services Vaults with full backup coverage.
DDE Firewall Vault — rsv-dde-firewall-backup:
| VM | Role | Status |
|---|---|---|
| ACTDIRAZC01 | Active Directory domain controller | ✅ Protected |
| ACTDIRAZC02 | Active Directory domain controller (secondary) | ✅ Protected |
| ENTRACAZC01 | Entra Connect sync server | ✅ Protected |
| Palo Alto firewall VM | DDE network security | ✅ Protected |
| Panorama VM | Palo Alto centralized management | ✅ Protected |
| Syslog VM | Arctic Wolf log forwarding | ✅ Protected |
| VLC VM | Arctic Wolf log collector | ✅ Protected |
Immutability: ✅ Enabled (Unlocked)
DDE Billings Vault — rsv-dde-billings-backup:
| Coverage | Details |
|---|---|
| Session hosts | 7 AVD session hosts |
| Retention | 30-day daily |
| Soft delete | ✅ Enabled |
| Immutability | ✅ Unlocked |
Veeam DR Replication (Azure → AWS)
Veeam replicates Azure production workloads nightly to the AWS DR environment. This is the recovery mechanism for a full Azure failover.
Configuration
| Setting | Value |
|---|---|
| Source | Azure production VMs (Production subscription) |
| Destination | AWS S3, Backup account (863609217450) |
| Frequency | Nightly |
| RPO | ~24 hours |
| S3 encryption | AWS KMS (customer-managed key) |
| Versioning | ✅ Enabled |
| Object Lock | ✅ Enabled |
| S3 lifecycle | Standard-IA at 90 days → Glacier at 365 days |
| Noncurrent version expiration | 6 years (HIPAA retention) |
March 4, 2026 Validation Results
| Metric | Result |
|---|---|
| Hosts replicated | 2/2 (CGIRDPAZP01, BIZARCAZP01) |
| Data processed | 106.4 GB |
| Data transferred | 11.6 GB (2.7× compression) |
| Duration | 5 hours 27 minutes |
| Transfer rate | 462 MB/s |
| Errors | 0 |
Recovery Process
During a DR failover, the Veeam backup template is attached to an EC2 instance in the AWS Production account (807267566999) and booted. See DR Failover Procedure for step-by-step instructions.
Data loss caveat: Data submitted since the last nightly backup must be re-submitted by customers after failover. This is a known accepted risk with RPO ~24 hours.
Backup Monitoring
Weekly HIPAA audit email: The ops-automation backup verification workflow runs weekly and includes vault job status in the HIPAA compliance audit email. Any backup job failures surface in the Monday morning report.
Azure Monitor alerts: Fire on backup job failures and missed jobs — routes to IT admin email.
Veeam: Sends job completion reports by email. Any replication failure increases RPO until the next successful run.
Restore Testing Schedule
| Quarter | Test | Scope |
|---|---|---|
| Q2 (June 2026) | Full DR failover | Veeam restore to AWS |
| Q3 (October 2026) | Recovery Vault restore | Azure VM restore from vault |
| Q4 (December 2026) | Backup coverage audit | Verify all critical resources are protected |
Compliance Relevance
| Requirement | Control | Implementation |
|---|---|---|
| HIPAA §164.308(a)(7)(ii)(A) — Data backup plan | Back up ePHI | Recovery Services Vault daily backups, Veeam nightly replication |
| HIPAA §164.308(a)(7)(ii)(B) — DR plan | Restore from backup | DR failover procedure using Veeam backups |
| HIPAA §164.308(a)(7)(ii)(D) — Testing | Test contingency plans | Veeam validated March 4, 2026; quarterly restore tests, June and October DR tests |
| SOC2 A1.2 — Recovery objectives | Define and test RTO/RPO | RPO ~24h documented, Veeam validated, RTO tracked during DR tests |
Document History
| Date | Change | Author |
|---|---|---|
| June 2026 | Veeam server rebuilt via Terraform on 2026-06-09 after license expiry (VM destroy/recreate; backup protection re-associated; vault data preserved). Backup facts in this doc re-verified against Terraform during the June review — no changes needed | Kobe |
| March 2026 | Removed DDE gap warning — DDE firewall vault (7 VMs) and billings vault (7 AVD session hosts) now fully deployed with soft delete + immutability Unlocked; added Veeam March 4 validation results; added S3 lifecycle policy details; added vault hardening section; added backup monitoring via HIPAA audit email | Rory |
| March 2026 | Replaced placeholder stub with full backup architecture documentation | Rory |