Docs

Appearance

Runbook: Offline Backup Procedure ​

Purpose ​

Some assets are important enough that a purely digital, online backup is not sufficient. This procedure defines the offline backup kit — the copy of last resort when every online system we own is compromised, encrypted, or otherwise unreachable.

Offline backup is the last layer of a defence-in-depth strategy. Online backups (Veeam, Azure Backup, S3 replication) catch most failures. Offline catches the scenarios that everything else cannot: full tenant compromise, ransomware that reaches the backup servers, coordinated insider destruction, or a Microsoft-scale platform event.

Scope — What Gets Backed Up Offline ​

The offline kit contains exactly three categories of artefact. Keep it minimal; every extra item is a new rotation burden.

1. Terraform state ​

Snapshots of the current .tfstate file for each repository — at the time of each quarterly refresh, not continuously. The offline copy is deliberately stale because its purpose is catastrophic recovery, not routine restore.

Repos included:

  • azure-infra
  • azure-dde-infra
  • aws-infra (one state file per AWS account — 7 files)
  • ops-automation
  • bedrock-hub
  • palo-alto-configs

File format: the raw .tfstate JSON, unencrypted on the USB (the USB is itself encrypted).

2. Break-glass credentials ​

The printed kit (stored in a tamper-evident envelope, separate from the USB but in the same safe):

  • 2 Entra PROD break-glass account credentials (username, password, MFA method)
  • 2 Entra DDE break-glass account credentials
  • 7 AWS root account credentials (one per account)
  • 3 on-prem AD domain admin break-glass credentials
  • 4 Palo Alto firewall local admin credentials
  • Hardware MFA tokens (FIDO2 keys and printed TOTP seed recovery codes)
  • Key Vault passphrase and private-key recovery material (for CMK bootstrap)
  • USB decryption passphrase (in a sealed envelope, separate from the USB)

3. Key contacts ​

A printed contact sheet:

  • Rory — role, personal cell, personal email, physical address
  • Kevin — T1 DR Domain Admin, personal cell, home address, Signal username
  • Greg — T1 DR Domain Admin, personal cell, home address, Signal username
  • Paul — customer relationship owner, personal cell, company landline
  • Adriana — BAA / vendor / risk, personal cell, company landline
  • Arctic Wolf MDR — 24x7 SOC number, account manager, portal URL, P1 escalation
  • IR Retainer (IRRET-006) — emergency number, account manager, onboarding contact
  • Microsoft Unified Support — contract number and break-glass contact
  • Legal counsel — primary and backup, with after-hours numbers
  • Insurance (cyber liability) — claims line and policy number

Also included on the sheet: the pre-agreed physical meeting location for full-digital- blackout scenarios (documented here by reference only — not repeated in this file).

Optional: runbook exports ​

Copies of the 10 most critical runbooks on the USB, so that if GitHub and bedrock-docs are unreachable we still have recovery procedures:

  • incident-response.md
  • out-of-band-comms-plan.md
  • annual-bgkit-verification.md
  • backup-architecture.md
  • common-tasks.md
  • DR failover procedure (when it exists in aws/)
  • crown-jewel-inventory.md
  • palo-alto-backup-credential-rotation.md
  • ad-recovery runbook (when it exists)
  • this document

Cadence — How Often ​

Offline backup runs quarterly — four times per year. This is a deliberate trade- off:

  • Too frequent (monthly) wastes Rory's time and risks print-and-shred errors.
  • Too rare (annual) leaves the offline copy badly out of date during an actual incident — a one-year-old Terraform state is an expensive starting point.
  • Quarterly matches the cadence of most credential rotations and major architectural changes.

Schedule:

QuarterMonthAligned with
Q1MarchAnnual review preparation
Q2JuneDecommission milestone (azure-infra target retirements)
Q3SeptemberMid-year compliance checkpoint
Q4DecemberYear-end — freeze state before holiday changes

Plus any unscheduled refresh triggered by:

  • Any break-glass credential rotation
  • Major architectural change (new AWS account added, new tenant, new crown jewel)
  • Addition or removal of a Signal OOB group member
  • Change of key contact (someone leaves, someone new on-call)

Media — What It Lives On ​

Two physical forms, stored together in the same safe.

Encrypted USB drive ​

  • Hardware: a FIPS-140-2 Level 3 encrypted USB drive (e.g. IronKey D500S, Apricorn Aegis Secure Key). Hardware encryption, not software — do not rely on BitLocker-on- standard-USB because we cannot audit the firmware.
  • Capacity: 32 GB or larger — ample for state files plus runbooks.
  • Two drives, rotating: at each quarterly refresh, write the new content to the drive that is NOT currently in the safe, verify it, then swap. The previous drive becomes the new rotation target and is wiped immediately after the swap. This guarantees that at any moment there is always a readable drive in the safe.
  • Passphrase: stored in a sealed envelope in the same safe, printed in block letters. Never emailed, never photographed, never typed into any computer connected to a network.
  • Drive identifier: write a unique serial number and the quarter on the drive's label (e.g. OB-2026-Q2-#01) so during an incident it is obvious which drive is current.

Printed credentials and contacts ​

  • Paper: archival-grade paper (100% cotton, acid-free) so the kit does not degrade over five-plus years of storage.
  • Envelopes: tamper-evident, numbered, signed-across-seal. Each envelope holds one account class (for example: "Entra PROD break-glass — sealed 2026-Q2").
  • Printer: a dedicated offline printer kept in Rory's office. Never on the corporate network. Print job log cleared after every use.
  • No photocopies: if a second copy is needed for Kevin or Greg, print a fresh one — do not photocopy, the photocopier has a hard drive and a memory.

Where stored ​

  • Primary safe: fireproof, UL-rated for at least 1 hour at 1700°F, secured in Rory's office or Cirius HQ secure storage. Combination access with Rory as primary holder and Kevin + Greg as secondary holders (see runbooks/annual-bgkit-verification.md for the access procedure).
  • Secondary safe (future consideration): a second location, offsite, for geographically-redundant storage. Not currently implemented — single-safe is the present state, a gap acknowledged.

Procedure — Quarterly Refresh ​

Allow 2 hours for a quarterly refresh. Do not rush.

Preparation (day before) ​

  1. Confirm the safe is accessible (Rory visits, opens, closes — no content changes yet).
  2. Pull the list of all crown-jewel backups, break-glass accounts, and contacts — reconcile against the printed inventory for changes in the last quarter.
  3. Print a draft of the new credential sheet and contact sheet on the offline printer. Visual-proof against last quarter's version. Red-pen any differences.
  4. Boot the trusted admin workstation (airgapped if possible) and prepare to export Terraform state.

Export Terraform state ​

For each repo listed in §1 above:

bash
# Example for azure-infra — adjust backend settings per repo
cd /home/ubuntu/src/azure-infra
terraform state pull > /tmp/offline-backup/azure-infra.tfstate

Repeat for every repo. Verify each file is valid JSON (jq empty <file>) before continuing.

Write USB ​

  1. Take the rotation USB (the one NOT currently in the safe).
  2. Plug into the trusted workstation. Unlock with the passphrase.
  3. Reformat the USB (the hardware reset that zeros the drive, not just a soft format).
  4. Create the directory structure:
    /terraform-state/
  azure-infra.tfstate
  azure-dde-infra.tfstate
  aws-infra-management.tfstate
  aws-infra-backup.tfstate
  aws-infra-dev.tfstate
  aws-infra-identity.tfstate
  aws-infra-logging.tfstate
  aws-infra-networking.tfstate
  aws-infra-prod.tfstate
  ops-automation.tfstate
  bedrock-hub.tfstate
  palo-alto-configs.tfstate
/runbooks/
  incident-response.md
  out-of-band-comms-plan.md
  annual-bgkit-verification.md
  backup-architecture.md
  common-tasks.md
  crown-jewel-inventory.md
  palo-alto-backup-credential-rotation.md
  dr-failover-procedure.md
  ad-recovery.md
  offline-backup-procedure.md
/contacts.txt
/README-recovery.md
/manifest.txt
  5. Generate a manifest.txt listing every file with its SHA-256 hash and line count or byte size. This is the integrity check future verifications will run against.
  6. Safely unmount the USB (flush writes).

Swap USB in safe ​

  1. Go to the safe. Open it.
  2. Take out the currently-resident USB. Set it aside — it will be wiped.
  3. Place the newly-written USB in the safe, in its labelled slot.
  4. Confirm the USB decryption passphrase envelope is still sealed and unmodified.
  5. Lock the safe.
  6. On the trusted workstation: wipe the now-removed previous USB (hardware zero).
  7. Label it with the next rotation date and store it with the workstation.
  1. For any account whose credential rotated since last quarter: print the new page on the offline printer.
  2. Place the new page in a fresh tamper-evident envelope; sign and date across the seal.
  3. Shred the old page in Rory's physical presence, same-day, cross-cut shredder.
  4. Return the new envelope to the safe.
  5. Update the rotation log sheet that lives inside the safe: date, account, who performed.

Contact-sheet refresh (only if changed) ​

Same procedure as credentials: print new, seal, shred old, file log.

Close-out ​

  1. Re-lock the safe.
  2. Record the quarterly refresh in bedrock-docs/compliance/annual-review-checklist.md (or the backup-specific log) — date, files written, size, passphrase envelope unmodified (y/n).
  3. Post an evidence record to SecOps via POST /api/evidence with control_id = "BGKIT-OFFLINE" and period_end = <refresh date>.
  4. Schedule the next quarterly refresh on the calendar.

Verification ​

Beyond the simple writing, the offline backup must be verified readable. This happens in two places:

At each quarterly refresh (informal) ​

After writing the new USB, before putting it in the safe, mount it on a different workstation and list the files. Open two or three at random. If any fail to open or show corruption, discard the USB and start over.

At annual break-glass kit verification (formal) ​

See runbooks/annual-bgkit-verification.md §4. The annual verification pulls the USB from the safe, mounts it, lists contents, checks freshness dates, and opens a sample of files. Any failure is a finding.

Recovery Use Case — When the Kit Is Opened for Real ​

The kit is opened in one of three scenarios:

Scenario A: Total Azure compromise, ransomware in backups ​

Symptom: Veeam catalog encrypted, Azure Backup restore-points all ransomed, Entra break-glass MFA not responding (the MFA provider is also Azure-hosted).

Response:

  1. OOB comms group activated per out-of-band-comms-plan.md.
  2. Open the safe. Retrieve printed break-glass credentials AND the USB.
  3. Log into Azure portal using the printed break-glass Entra account and the hardware FIDO2 key from the safe (which does not depend on Azure).
  4. Stand up a clean recovery subscription.
  5. Restore Terraform state from the USB — one repo at a time, starting with azure-infra. Use the printed runbook copies from the USB to follow the procedure (GitHub may also be unreachable).
  6. Run terraform plan against the restored state — expect drift, accept one day of reconciliation.
  7. Rebuild AD, Key Vault, SecOps, in that order (see security/crown-jewel-inventory.md recovery order).

Scenario B: Individual item recovery ​

Symptom: a specific account's credential is lost (not stolen — just lost) and online password reset is failing for some reason.

Response:

  1. Standard break-glass procedure — open safe, retrieve the specific credential, use it, then rotate and re-seal. Do NOT use the offline USB unless the online path is actually compromised.

Scenario C: Annual verification ​

Symptom: it's time for the annual verification. The USB is opened per annual-bgkit-verification.md §4, not as an incident but as a drill. After verification, re-seal and return.

Do Not Do ​

  • Never photograph the printed credentials, not even "just for a minute".
  • Never email any content from the offline kit.
  • Never type the USB decryption passphrase into a networked computer.
  • Never store the passphrase and the USB in the same envelope.
  • Never skip a quarterly refresh — the kit becomes less useful with every quarter of drift.
  • Never extend the scope of the kit casually — every new item is a new rotation burden and a new thing that can go stale.
  • Never reuse an old USB without hardware-wiping it first.

Document History ​

DateChangeAuthor
April 2026BGKIT-010: Initial offline backup procedure — quarterly cadence, encrypted USB, printed credentials, physical safeKobe

Internal use only — Cirius Group