Backup Restore Verification

Purpose

Periodic verification that backups are intact and restorable — before you need them. This is distinct from the actual restore procedure (Veeam Restore Procedure) and from the full DR failover test (DR Failover Procedure).

HIPAA requirement: §164.308(a)(7)(ii)(A) — data backup plan must be tested to verify restorability. Evidence goes in the DR Test Results Log.

Frequency: Quarterly, or before any planned DR test.
Owner: Rory

Verification Schedule

Check	Frequency	Evidence
RSV job status (all vaults)	Weekly — automated via HIPAA audit email	Email report
Veeam nightly job completion	Weekly — automated via Veeam email report	Email report
S3 Object Lock verification	Quarterly	Screenshot of bucket policy
RSV soft delete + immutability	Quarterly	Portal screenshot
File-level test restore	Quarterly (Q2 DR test)	Restore timestamp + screenshot
Full Veeam VM boot test	Quarterly (Q2/Q4 DR tests)	Boot screenshot, RTO timer

1 — RSV Job Status

What to Check

All Azure Recovery Services Vaults must show recent successful backup jobs. A missed backup is a gap in the RPO window.

Automated Check (Weekly)

The HIPAA audit email (sent weekly by ops-automation) includes a backup job summary. Review it for:

Any vault showing Failed or Warning status
Any vault with no job in the last 25 hours (missed run)

If the email hasn't arrived in over 7 days, check the ops-automation GitHub Actions for the weekly-hipaa-audit.yml workflow.

Manual Check

bash

# Check all backup jobs that are NOT in Completed status
az backup job list \
  --resource-group rg-prod-backup \
  --vault-name rsv-prod-backup \
  --output table \
  --query "[?properties.status!='Completed']"

Repeat for each vault:

Vault	Resource Group
`rsv-prod-backup`	`rg-prod-backup`
`rsv-identity-backup`	`rg-identity`
`rsv-firewall-backup`	`rg-firewall`
`rsv-dev-backup`	`rg-dev`
`rsv-dde-firewall-backup`	`rg-dde-firewall`
`rsv-dde-billings-backup`	`rg-dde-billings`

Any failed job needs investigation. Check the job details in the portal:
Azure Portal → Recovery Services Vault → Backup Jobs → click the failed job

Common causes:

VM was deallocated during the backup window
Snapshot limit reached (>512 recovery points) — delete old snapshots
Network timeout — usually self-resolves on next run

2 — Veeam Nightly Job

Veeam runs nightly replication from Azure Production to the AWS Backup account (863609217450), covering CGIRDPAZP01 and BIZARCAZP01.

What to Check

Veeam sends an email completion report after each run. The report shows:

Job name, status (Success / Warning / Failed)
Data size processed and transferred
Duration
Any VMs that failed to replicate

If no Veeam email has arrived in 25+ hours: replication failed silently or the Veeam server is down.

Investigating a Veeam Failure

Access the Veeam management console on the Veeam server (Production subscription — access via Twingate + RDP)
Home → Jobs → Backup Copy — check the last run status
Click the failed job → Statistics — shows per-VM errors
Common fixes:
- AWS connectivity error: check the Veeam server's outbound connectivity to AWS S3; verify the IAM credentials used by Veeam haven't expired
- Insufficient storage: check the S3 bucket has available capacity (it should, with lifecycle rules moving to IA/Glacier)
- VM snapshot failure: VM may have been shut down or snapshotted by another process — retry the job

3 — S3 Object Lock Verification (Quarterly)

Veeam backup data in the AWS Backup account is protected by S3 Object Lock. Quarterly, verify Object Lock is still enforced — this protects backups from ransomware that gains AWS credentials.

Check Object Lock Configuration

bash

# Verify Object Lock is enabled on the Veeam backup bucket
aws s3api get-object-lock-configuration \
  --bucket <veeam-backup-bucket-name> \
  --profile backup \
  --output json

Expected output includes:

json

{
  "ObjectLockConfiguration": {
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Days": 30
      }
    }
  }
}

COMPLIANCE mode means even root cannot delete objects within the retention window. If the mode shows GOVERNANCE or is missing, escalate — this is a critical finding.

Verify a Sample Object is Locked

bash

# Check the lock on a specific backup object
aws s3api get-object-retention \
  --bucket <veeam-backup-bucket-name> \
  --key <path-to-backup-file> \
  --profile backup

Returns the retain-until date. If the date is in the future, the object is protected.

Take a screenshot of both outputs for compliance evidence.

4 — RSV Soft Delete and Immutability (Quarterly)

Azure Recovery Services Vaults should have soft delete and immutability enabled. Soft delete retains deleted backup data for 14 days; immutability prevents deletion of backup data within the retention window.

Check Soft Delete Status

Azure Portal → Recovery Services Vault → Properties → Security Settings

Soft delete should show Enabled. If disabled, someone turned it off — re-enable immediately and investigate who disabled it via Azure Activity Log.

bash

# Check via CLI
az backup vault backup-properties show \
  --name rsv-prod-backup \
  --resource-group rg-prod-backup \
  --query "properties.softDeleteFeatureState"

Expected: "Enabled"

Check Immutability Status

Azure Portal → Recovery Services Vault → Properties → Immutability

Should show Unlocked (enabled but not permanently locked — appropriate for active vaults where you may need to adjust policy). Never lock permanently on active production vaults — locking is irreversible and prevents policy changes.

5 — File-Level Test Restore (Quarterly)

A quarterly file restore proves the backup is actually usable — not just that the job ran successfully.

Procedure

Pick a non-critical VM from the Production RSV — a server that is not customer-facing and can tolerate a brief additional load
Azure Portal → rsv-prod-backup → Backup Items → Azure Virtual Machine → [target VM]
Click File Recovery (not Restore VM)
Select the most recent recovery point
Click Download Script — note the date/time of the recovery point and the password
Run the script on any Windows machine with Twingate access to the vault
Confirm the backup disk mounts successfully as a drive letter
Navigate to a known file (e.g., C:\Windows\System32\drivers\etc\hosts — always present)
Verify the file is readable
Click Unmount Disks in the portal
Record: test date, VM used, recovery point date, result (Success / Failure)

Evidence to Capture

Screenshot of the mounted drive in Windows Explorer
Screenshot of the recovery point selected
Timestamp of when the mount succeeded

Add findings to the DR Test Results Log.

6 — Veeam VM Boot Test (Q2 / Q4 DR Tests)

A full Veeam VM boot test is part of the Q2 and Q4 DR tests — not done independently. For the full procedure see DR Failover Procedure.

Verification target: at least 3 tier-1 workloads must boot cleanly from Veeam replica with no manual intervention.

The tier-1 workloads are:

CGIRDPAZP01 — DR domain controller
BIZARCAZP01 — business application server
Any third workload specified in the Q2/Q4 test plan

7 — PostgreSQL Database (SecOps Platform)

The SecOps platform database (psql-secops-prod) uses Azure Database for PostgreSQL with automated PITR (point-in-time restore) enabled.

Verify PITR is Enabled

Azure Portal → psql-secops-prod → Overview

Check "Backup retention period" — should be 7 days minimum. The earliest restore point shown must be within the configured retention.

bash

az postgres flexible-server show \
  --name psql-secops-prod \
  --resource-group rg-logging-logs \
  --query "{backupRetentionDays:backup.backupRetentionDays, geoRedundantBackup:backup.geoRedundantBackup}"

Test PITR Restore (Annual, During DR Test)

During the annual DR test, verify PITR works by restoring to a test server:

bash

az postgres flexible-server restore \
  --resource-group rg-logging-logs \
  --name psql-secops-pitr-test \
  --source-server psql-secops-prod \
  --restore-time "2026-05-24T01:00:00Z"

Connect to the restored server, verify the data looks correct for the target timestamp, then delete the test server:

bash

az postgres flexible-server delete \
  --resource-group rg-logging-logs \
  --name psql-secops-pitr-test \
  --yes

What to Do If Verification Fails

Finding	Action
RSV backup job failed	Investigate and resolve; check next day's run; document if gap occurred
Veeam didn't run in 25h	Access Veeam console, check job status, fix connectivity or credentials
S3 Object Lock missing or GOVERNANCE mode	CRITICAL — escalate to Rory immediately; do not delete any objects
RSV soft delete disabled	Re-enable; check Activity Log for who disabled it; treat as security event
File restore fails (corrupt or inaccessible)	Open issue; test a different recovery point; if all points fail, escalate
PITR restore fails	Check Azure Status for PostgreSQL service issues; open support ticket if needed

All failures are logged in the DR Test Results Log as open gaps with severity, owner, and remediation status.

HIPAA Evidence Package

For §164.308(a)(7)(ii)(A) — backup plan testing:

This document — the verification procedure
Weekly HIPAA audit emails — RSV job status automated checks
Veeam email reports — nightly replication confirmation
Quarterly verification screenshots — S3 Object Lock, RSV immutability, file restore
DR Test Results Log — formal test outcomes including RTO/RPO achieved
DR Test Plan documents (Q2/Q4) — shows structured scope

Backup Architecture — what is backed up, schedules, retention
Veeam Restore Procedure — actual restore steps when recovery is needed
DR Failover Procedure — full environment failover
DR Test Results Log — evidence log for HIPAA audit

Document History

Date	Change	Author
May 2026	Initial draft — weekly/quarterly verification checks for RSV, Veeam, S3 Object Lock, RSV immutability, file restore, PostgreSQL PITR. HIPAA evidence mapping.	Rory

Backup Restore Verification ​

Purpose ​

Verification Schedule ​

1 — RSV Job Status ​

What to Check ​

Automated Check (Weekly) ​

Manual Check ​

2 — Veeam Nightly Job ​

What to Check ​

Investigating a Veeam Failure ​

3 — S3 Object Lock Verification (Quarterly) ​

Check Object Lock Configuration ​

Verify a Sample Object is Locked ​

4 — RSV Soft Delete and Immutability (Quarterly) ​

Check Soft Delete Status ​

Check Immutability Status ​

5 — File-Level Test Restore (Quarterly) ​

Procedure ​

Evidence to Capture ​

6 — Veeam VM Boot Test (Q2 / Q4 DR Tests) ​

7 — PostgreSQL Database (SecOps Platform) ​

Verify PITR is Enabled ​

Test PITR Restore (Annual, During DR Test) ​

What to Do If Verification Fails ​

HIPAA Evidence Package ​

Related Documents ​

Document History ​

Backup Restore Verification

Purpose

Verification Schedule

1 — RSV Job Status

What to Check

Automated Check (Weekly)

Manual Check

2 — Veeam Nightly Job

What to Check

Investigating a Veeam Failure

3 — S3 Object Lock Verification (Quarterly)

Check Object Lock Configuration

Verify a Sample Object is Locked

4 — RSV Soft Delete and Immutability (Quarterly)

Check Soft Delete Status

Check Immutability Status

5 — File-Level Test Restore (Quarterly)

Procedure

Evidence to Capture

6 — Veeam VM Boot Test (Q2 / Q4 DR Tests)

7 — PostgreSQL Database (SecOps Platform)

Verify PITR is Enabled

Test PITR Restore (Annual, During DR Test)

What to Do If Verification Fails

HIPAA Evidence Package

Related Documents

Document History