Appearance
DR Recovery Priority Inventory
Distinct purpose from the Threat Model Crown Jewels. This document lists assets by disaster-recovery priority — backup schedule, recovery order, RTO, RPO, and restore procedures. It is NOT the threat-model crown jewel inventory (which ranks assets by security impact-if-compromised). The threat-model crown jewel list lives exclusively in §2 of the Cirius Group Threat Model and uses a different set (CJ-1 through CJ-8) with different framing. Do not treat this document as a competing or alternative crown-jewel list.
Purpose
This document identifies the systems and datasets that must survive a ransomware event and defines their recovery priority. If we lose everything else, we can rebuild from Terraform. If we lose any of these, either the business is over (no customer delivery) or the next 48 hours of recovery are qualitatively worse (no TF state, no audit evidence, no way back into Azure).
This inventory drives three decisions:
- Backup priority. Crown jewels get offline, offsite, immutable backups that are physically and logically disconnected from any attacker path.
- Recovery order. During DR, restore in the order listed — dependencies first.
- Blast-radius testing. Every quarterly DR exercise must prove recovery of at least one crown jewel.
Everything else can be rebuilt from Terraform. These cannot.
Scope
Seven crown jewels, grouped by the type of loss they represent:
| # | Asset | Loss type | Recovery priority |
|---|---|---|---|
| 1 | DDE AVD environment | Business-ending — customer delivery stops | P0 |
| 2 | Active Directory (on-prem and Entra ID) | Authentication root of trust | P0 |
| 3 | Terraform state (all repos) | Cannot rebuild infra without it | P1 |
| 4 | psql-secops-prod database | SecOps incident history, known-good rules, CM tickets | P1 |
| 5 | cirius-openai-kv-prod Key Vault | Service-principal secrets, API keys, rotation keys | P1 |
| 6 | Veeam backup catalog | Knowing which backup maps to which VM | P1 |
| 7 | cirius-audit-evidence-prod S3 bucket | Six-year HIPAA audit evidence archive | P2 |
1. DDE AVD Environment
Why it matters
DDE (ciriusdde.com, tenant ff1c5d68) hosts the customer-facing Azure Virtual Desktop environment that delivers our published application to Medicare. If DDE is encrypted or deleted:
- Medicare customers cannot run the application — revenue stops, contractual SLAs breach within hours.
- Reconstituting from scratch requires rebuilding the image, reauthorising Medicare connectivity (days to weeks), and re-enrolling every customer session.
- This is the single most business-critical asset in the environment.
Where it lives
- Azure tenant: ciriusdde.com / ff1c5d68
- Host pool: AVD host pool in the DDE subscription
- Image: custom published-application image in the DDE compute gallery
- Networking: DDE spoke VNet peered to Palo Alto hub, customer traffic ingress via Application Gateway
- IaC repo: azure-dde-infra
Backup and recovery mechanism
- Image backup: custom image snapshots to Recovery Services Vault in the DDE subscription, soft delete and immutability (Unlocked) enabled, retained 30 days
- Config backup: Terraform state backs the entire VNet/host pool/gateway, plus the image definition — rebuilding from Terraform + image snapshot is the primary path
- Cross-tenant replication: no — DDE is intentionally isolated; a Palo Alto compromise in PROD must not bleed into DDE
- Offline backup: quarterly export of the image manifest and host-pool config to the encrypted USB in the physical safe (see
runbooks/offline-backup-procedure.md) - Recovery steps:
- Rebuild VNet and peering from azure-dde-infra Terraform
- Restore custom image from RSV snapshot
- Rebuild host pool against restored image
- Validate one session end-to-end before reconnecting Medicare ingress
- Reconnect App Gateway and notify customers
- RTO: 8 hours for full session restoration
- RPO: 24 hours (nightly image export)
Known risks
- DDE runs on the same Palo Alto hub as PROD — a hub compromise propagates. Mitigation is segmentation rules reviewed quarterly.
- Veeam does NOT back up AVD host pool config directly — the RSV image snapshot plus Terraform is the full recovery picture.
2. Active Directory — On-Prem AD and Entra ID
Why it matters
Every other system authenticates against AD. Kerberos tickets, SAML tokens, OIDC federated credentials, Conditional Access policies, device compliance state — all derived from AD and Entra ID. If AD is destroyed or held for ransom:
- Every domain-joined VM (including the Cloud PC, all Windows servers, BIZWRK* series) is unauthenticated.
- Entra-federated applications fail — Microsoft 365, GitHub OIDC, SecOps platform SSO, Twingate, everything.
- Rebuilding AD from scratch is weeks of work and assumes you can prove to Microsoft and third parties that the tenant is you.
Where it lives
- On-prem AD: two domain controllers, ACTDIRAZP01 (primary) and ACTDIRAZP02 (secondary), in the PROD Azure tenant. NTDS.dit and SYSVOL on both.
- Entra ID PROD: tenant d477c9f8, ciriusgroup.com
- Entra ID DDE: tenant ff1c5d68, ciriusdde.com
- Federation: on-prem AD sync to Entra PROD via Entra Connect (no ADFS)
- Privilege tiering: Tier 0 admin accounts in both directions, break-glass unsuppressible
Backup and recovery mechanism
- On-prem AD:
- Veeam backs up both DCs nightly with application-aware processing (VSS writers for AD) — NTDS.dit consistency guaranteed
- System State backup (Azure Backup, in-guest agent) nightly, separate RSV
- Restorable via Veeam full-VM restore OR System State restore on replacement VM
- Authoritative restore playbook lives in
runbooks/ad-recovery.md(to be written if missing)
- Entra ID:
- Microsoft 365 does not "back up" Entra in the traditional sense — rely on soft delete (30 days) for users, groups, app registrations
- Conditional Access policies, named locations, Entra Connect configuration exported weekly to the offline backup USB (see offline backup procedure)
- Break-glass accounts (2 PROD, 2 DDE) stored in the physical safe — guarantee access even if everything else is gone
- Break-glass escape hatch:
- If both DCs and Entra are unreachable, the break-glass Entra accounts in the printed kit still authenticate to Azure portal with Global Admin — sufficient to rebuild
- RTO: 4 hours for on-prem AD from Veeam, 2 hours for Entra via break-glass
- RPO: 24 hours for on-prem (nightly Veeam), near-zero for Entra (live service)
Known risks
- Entra Connect sync server is a privileged pivot point — compromise equals domain compromise. Hardened per
security/lpriv-baseline.md. - Microsoft's side of Entra is not under our control — an outage of Entra is a platform event, not a recovery scenario we can fix.
3. Terraform State — All Repos
Why it matters
Terraform state files describe what exists in Azure and AWS and what Terraform will do next. Without state, every terraform apply is a destroy-and-recreate, every running resource is orphaned, and rebuilding from .tf files cannot track drift. In a ransomware scenario where Azure itself is intact but state is lost, the recovery takes days of importing rather than minutes of applying.
Where it lives
- azure-infra: Azure Storage container in the PROD subscription, CMK-encrypted
- azure-dde-infra: Azure Storage in the DDE subscription, CMK-encrypted
- aws-infra: S3 bucket per account (7 buckets total), versioning on, MFA delete on critical buckets
- ops-automation: Azure Storage in PROD
- bedrock-hub: Azure Storage in PROD
- palo-alto-configs: Azure Storage in PROD
Backup and recovery mechanism
- Soft delete + versioning: every state backend has versioning / soft delete enabled for ≥30 days. A single corrupt apply is recoverable.
- Cross-region replication: Azure state containers replicate to paired regions (RA-GRS), AWS state S3 replicates to backup account (863609217450)
- Quarterly offline export: Rory copies the current
.tfstatefile for each repo to the encrypted USB in the physical safe — perrunbooks/offline-backup-procedure.md - Nightly pipeline snapshot: CI workflow exports state to a dedicated audit S3 bucket under WORM for 6-year retention
- Recovery steps:
- If recent (days-old) version exists in backend, restore via
az storage blob undeleteor S3 version rollback - If backend is destroyed, recover from the replicated backup account
- If nothing else works, restore the quarterly offline snapshot and accept that Terraform will
plansignificant drift — expect a day of reconciliation
- If recent (days-old) version exists in backend, restore via
- RTO: 1 hour from backend version history, 4 hours from cross-region replica, 12 hours from offline
- RPO: near-zero for versioned backend, 24 hours for replica, one quarter for offline
Known risks
- Never store state locally —
.terraform/terraform.tfstatemust stay empty. CI enforces this. - MFA delete on S3 buckets is strongly recommended but only enabled on
aws-infrastate buckets today — other buckets are a gap.
4. psql-secops-prod — SecOps PostgreSQL Database
Why it matters
The SecOps platform database holds:
- Every incident ever raised (with all evidence links, decisions, timeline)
- Known-good rules — the suppression logic the analyst agent depends on
- Stories, epics, project tracking (including this very story)
- Change management tickets created from every merged PR
- Evidence records tying files in S3 to HIPAA controls
- Audit trail of who did what, when — required for HIPAA accountability
Lose this and we lose the history of how the environment got to where it is. Audit evidence breaks. Known-good rules disappear, so the analyst agent starts generating false positives for every benign pattern we've ever curated. The six-year HIPAA retention requirement cannot be met from logs alone — the index is in the database.
Where it lives
- Azure Database for PostgreSQL Flexible Server: psql-secops-prod
- Resource group: rg-logging-logs
- Subscription: logging
- CMK-encrypted at rest, TLS 1.2 in transit, private endpoint only (no public)
Backup and recovery mechanism
- Automated backups: Azure PostgreSQL Flexible Server built-in backups, geo- redundant, 35-day retention (point-in-time restore)
- Long-term retention (LTR): weekly full backups retained 7 years via Azure Backup for PostgreSQL — HIPAA six-year bar met
- Logical dump: nightly
pg_dumpto the encrypted audit evidence bucket in AWS (cirius-audit-evidence-prod) — crosses the cloud boundary so an Azure-only event does not destroy the index - Known-good rules export: in addition, known-good rules are exported weekly to a JSON artefact stored on the offline USB backup — because they are the hardest to rebuild from incidents alone
- Recovery steps:
- For recent corruption: point-in-time restore to a new server, swap the private endpoint
- For full loss within LTR window: restore from LTR full backup
- For a cross-cloud event: restore from the AWS pg_dump (most recent nightly)
- RTO: 2 hours for point-in-time, 6 hours for LTR, 12 hours from pg_dump
- RPO: near-zero for point-in-time (WAL retained), 24 hours for pg_dump
Known risks
pgauditis not yet enabled on psql-secops-prod — row-level audit trail is limited to application logs; this is on the roadmap (Tier 1 Weekly — #13 pgaudit on psql-secops-prod).- pg_dump secret is stored in Key Vault (cirius-openai-kv-prod); loss of KV affects the pg_dump schedule indirectly.
5. cirius-openai-kv-prod — Azure Key Vault
Why it matters
Key Vault holds every secret that isn't a break-glass credential:
- Azure service principal client secrets used by CI/CD (github-deploy-main)
- INGEST_API_KEY used by every ops-automation script to talk to SecOps
- Palo Alto API keys, Panorama API keys, SES SMTP credentials
- Cortex XDR API tenants, Arctic Wolf integration tokens
- Backup-system service accounts, Veeam credentials
- pg_dump service account for psql-secops-prod
- Customer-managed encryption keys (CMK) for storage accounts, disks, databases
A compromised KV equals a compromise of the entire automation fabric. A destroyed KV equals every service principal rotation and every pipeline broken — weeks of reconstruction.
Where it lives
- Resource: cirius-openai-kv-prod, Standard tier
- Resource group: rg-logging-logs or rg-openai (verify at inventory time)
- Subscription: PROD
- Access: Azure RBAC, no access-policy model, private endpoint preferred for new consumers
- Soft delete: 90 days; purge protection enabled
Backup and recovery mechanism
- Soft delete: purge protection is ON, so deleted secrets are recoverable for 90 days and cannot be force-purged
- Managed HSM key backup: CMK keys backed up to a second KV in AWS Secrets Manager (cross-cloud redundancy for the keys used by CMK-encrypted storage)
- Weekly secret export: a curated export of non-rotating secret metadata (name, version, rotation date — NOT the secret value) to the encrypted offline USB
- Break-glass parallel: the break-glass accounts do not depend on KV — they are on paper in the safe
- Recovery steps:
- For accidental delete:
az keyvault secret recover --name ... - For KV deletion (blocked by purge protection for 90 days): recover the vault
- For full loss after purge window (should never happen): rotate every service principal and API key, using the break-glass identity to authenticate to Azure and bootstrap
- For accidental delete:
- RTO: minutes for soft-delete recovery; days to weeks for full rebuild
- RPO: real-time for secrets (every write persists); 7 days for the offline metadata export
Known risks
- The ops-automation CLAUDE.md instructs fetching INGEST_API_KEY via
az keyvault secret showat runtime — if KV is down, every ops-automation script fails open (logs and continues). Verify failure mode during DR. - Cross-cloud key backup (HSM → AWS) is partial today; complete coverage is on the roadmap.
6. Veeam Backup Catalog
Why it matters
Veeam backs up VMs to the AWS Backup account (863609217450) — but the catalog (the index that maps backup timestamps to VMs and consistency points) lives alongside the Veeam server. Without the catalog, the backups are an opaque blob store and recovery requires manual byte-level inspection.
Destroy Veeam catalog without destroying backups: recovery still possible but takes days of index reconstruction. Destroy Veeam catalog AND backups: complete loss of recovery capability for every Windows VM.
Where it lives
- Veeam server VM: separate from the protected workloads, in a dedicated management subnet
- Repository: AWS S3 in the Backup account (863609217450), immutability enabled on recent restore points
- Catalog: on the Veeam server's local disk (Veeam SQL database)
Backup and recovery mechanism
- Catalog backup: Veeam's built-in "Configuration Backup" writes the SQL catalog to the S3 repository daily — so the catalog itself is one of the backups
- Veeam server VM-level backup: the Veeam server is itself backed up by Veeam (bootstrap chicken-and-egg problem handled via quarterly out-of-band copy)
- Quarterly export: the Veeam Configuration Backup file is copied to the offline USB in the physical safe — this is the last-line break-glass recovery for the catalog
- S3 immutability: recent Veeam restore points are Object-Lock in compliance mode, cannot be deleted even by root account
- Recovery steps:
- Rebuild Veeam server from VM backup (or stand up fresh Veeam server)
- Restore Configuration Backup from S3 repository → catalog restored
- Verify backup integrity against a known VM recovery test
- Proceed with VM recovery per incident runbook
- RTO: 4 hours for catalog restore (prerequisite for all other Veeam recovery)
- RPO: 24 hours for catalog; same as parent backup schedule for VMs
Known risks
- The AWS Backup account root credentials are a break-glass crown jewel by themselves — they control the S3 bucket holding every VM backup. Any compromise of that root account is a P0 incident.
- Immutability is only enabled on recent restore points — long-term archive restore points are mutable, a gap to close.
7. cirius-audit-evidence-prod — HIPAA Audit Evidence Bucket
Why it matters
HIPAA requires six years of audit evidence retention. This bucket holds:
- Annual penetration test reports
- Maester M365 weekly audit outputs
- Azure Policy compliance snapshots
- SecurityHub HIPAA standard snapshots
- Change management tickets exported from SecOps
- Break-glass kit annual verification logs
- pg_dump of psql-secops-prod (see crown jewel #4)
Lose this bucket and we lose the ability to respond to a HIPAA audit. Six years of control evidence gone means OCR finds us in violation of the Security Rule's documentation requirement, which carries corrective-action-plan and monetary penalties.
Where it lives
- AWS S3 bucket: cirius-audit-evidence-prod
- Account: Logging (038901680748)
- Region: us-east-1
- Configuration: versioning enabled, Object Lock compliance mode, KMS CMK encryption, MFA delete enabled, public access blocked at bucket + account level
Backup and recovery mechanism
- Object Lock compliance mode: objects cannot be deleted or overwritten for the retention period, not even by the account root. This is the primary durability guarantee.
- Cross-region replication: replicates to a second bucket in us-west-2, same account, same compliance-mode lock
- Versioning: every write creates a new version; previous versions retained for the full retention period
- Inventory reports: S3 Inventory generates a weekly CSV of every object, stored in a separate operations bucket — this is the "catalog" so we can find evidence during an audit without scanning the main bucket
- Recovery steps:
- For accidental delete: recover via version history
- For region loss: read from cross-region replica
- For full bucket loss (would require Object Lock bypass, which compliance mode blocks): not recoverable — the design intent is that this never happens
- RTO: minutes for version restore, hours for region failover
- RPO: near-zero (synchronous versioning)
Known risks
- KMS CMK deletion would render all objects unreadable. Key deletion is protected by a 30-day pending-deletion window and requires MFA — but this is still the single theoretical point of failure.
- Retrieval cost from S3 during a large audit is non-trivial; budget for it separately.
Dependencies and Recovery Order
Restoring crown jewels has ordering constraints:
- Active Directory first (both on-prem DCs and Entra via break-glass if needed) — every other system authenticates against it.
- Key Vault second — service principals depend on KV secrets to function.
- Terraform state third — needed to rebuild any other infrastructure.
- psql-secops-prod fourth — once there is infra to host it, the DB gives back incident history and known-good rules.
- Veeam catalog fifth — needed before restoring any VM-level backups.
- DDE AVD sixth — depends on all the above (AD for auth, TF state for infra).
- cirius-audit-evidence-prod last — passive archive, not on the critical path for restoring business operations; can be verified after everything else is up.
This ordering is the contract for quarterly DR exercises.
Document History
| Date | Change | Author |
|---|---|---|
| April 2026 | BGKIT-009: Initial crown jewel inventory — 7 assets with backup/recovery mechanisms and RTO/RPO | Kobe |