Runbook: Annual Break-Glass Kit Verification

Purpose

Annual verification that the physical break-glass kit actually works when we need it — when Entra is compromised, the digital password manager is unreachable, and the normal communication channels cannot be trusted.

This is not a paperwork exercise. The kit has failed before in other organisations because nobody tried it until the day it was needed: safes forgotten open, printed credentials rotated but never reprinted, Signal accounts with expired session keys, USB sticks unreadable. The ransomware event in November 2024 proved that the unhappy path is the path we will actually walk. Verify the kit every year.

Scope

The break-glass kit covers 18 accounts plus offline artefacts:

• 3 on-prem Active Directory domain admin accounts
• 7 AWS account root credentials (Management, Backup, Dev, Identity, Logging, Networking, Prod)
• 2 Entra PROD break-glass (cirius-breakglass@ciriusgroup.com and the PROD tenant secondary)
• 2 Entra DDE break-glass (cirius-breakglass@ciriusdde.com and the DDE tenant secondary)
• 4 Palo Alto local admin accounts (one per firewall)
• Printed key contact list (Rory, Kevin, Greg, Paul, Adriana, Arctic Wolf, IR Retainer)
• Encrypted USB with Terraform state backup and critical runbook exports
• Signal OOB comms group membership

Cadence

Activity	Frequency	Owner
Full annual verification (this runbook)	Once per calendar year	Rory
Safe access test for Kevin and Greg	Once per calendar year (same visit)	Kevin, Greg
Printed credential refresh (if any rotated)	Immediately after rotation	Rory
Signal group ping	Quarterly (separate, lighter)	Rory

Prerequisites

Before starting the annual verification:

• Block a 2-hour window on the calendar. The break-glass kit test is disruptive if rushed.
• Confirm the physical safe combination still works for Rory before the visit (no surprises in front of witnesses).
• Pull the current list of break-glass account usernames from Entra, AWS, Palo Alto, and on-prem AD. You need this to compare against the printed kit.
• Confirm Kevin and Greg are both available for their portion — do not do their verification without both of them.
• Notify SecOps: any break-glass account activity is CRITICAL and unsuppressible. Expect alerts when you use the printed credentials.

---

1. Test Break-Glass Login from Printed Credentials Only

This step simulates the real emergency: digital password manager, Entra SSO, and MFA are all assumed unavailable. You must log in using only what is printed on paper.

For each account class, follow this sequence:

1a. Entra PROD break-glass (cirius-breakglass@ciriusgroup.com)

1. Use a personal device or a clean browser profile — do NOT log in on any device with cached sessions.
2. Navigate to https://portal.azure.com in a private/incognito window.
3. Enter the username exactly as printed.
4. Enter the password exactly as printed. Type it — do not paste.
5. Complete MFA using the hardware token stored in the safe (FIDO2 key or printed TOTP seed backup). Do NOT use Authenticator app — the whole point is that the phone might be compromised or lost.
6. Confirm login succeeds and the account retains Global Admin role.
7. Immediately sign out and close the browser. Do not click around.
8. Record: did login succeed? Did MFA work from the physical token? Record timestamp.

1b. Entra DDE break-glass (cirius-breakglass@ciriusdde.com)

Repeat 1a against the DDE tenant. Same private browser approach. Same sign-out discipline.

1c. AWS root — spot check two of seven

Testing all seven AWS root logins is invasive. Spot-check two:

• Management root (account 206820231356) — mandatory every year
• One other, rotated annually: pick in the order Backup → Logging → Networking → Prod → Identity → Dev → Management, wrapping yearly.

Procedure:

1. Navigate to https://console.aws.amazon.com in a private browser window.
2. Choose "Root user", enter the printed email and password.
3. Complete MFA using the hardware key stored in the safe (one of the two yubikeys registered for root).
4. Confirm login succeeds and billing view is accessible.
5. Sign out immediately.

1d. On-prem AD domain admin — spot check one of three

Pick one of the three on-prem AD domain admin break-glass accounts on a rotating basis.

1. From the console of CGIRDPAZP01 or an ADDS-joined admin workstation (domain admin station only — do NOT use a user workstation).
2. Log off current session.
3. Log in using the printed on-prem AD break-glass username and password.
4. Run whoami /groups and confirm Domain Admins membership.
5. Sign out immediately.

1e. Palo Alto local admin — spot check one of four

Spot-check one firewall local admin per year, rotating through PA-01 → PA-02 → PA-03 → PA-04.

1. Connect to the firewall management interface via the management VLAN jump host.
2. Log in using the printed local admin username and password (not the Panorama-managed admin — the firewall-local fallback).
3. Run show system info and confirm you can see the CLI.
4. Log out immediately.

1f. Summary table for the verification log

Record results in the verification log (append to bedrock-docs/compliance/annual-review-checklist.md or the dedicated audit evidence folder):

Account class	Account tested	Login result	MFA method	Notes
Entra PROD BG	cirius-breakglass@ciriusgroup.com
Entra DDE BG	cirius-breakglass@ciriusdde.com
AWS root — Management
AWS root — rotation pick
On-prem AD DA
Palo Alto local admin

Any failure is a blocker — fix it before closing the annual review.

---

2. Physical Safe Access Test — Kevin and Greg

Kevin and Greg are the T1 Domain Admin (DR) escalation path. If Rory is unavailable during an incident they must be able to reach the printed kit. This step confirms they can.

Do this step with both Kevin and Greg present. Do not do it for them — have them do it themselves with Rory observing.

1. Rory arrives at the safe location first and does NOT open the safe.
2. Kevin opens the safe using his credentials (combination, key, or dual-control token as configured). Record: did the combination work on the first attempt?
3. Kevin reads one item from the printed kit aloud (pick a non-sensitive entry — a phone number, a label, the date the kit was last refreshed). This proves he can read it, not just touch it.
4. Kevin closes the safe.
5. Greg repeats the same sequence independently. Do not coach him — he either remembers his access procedure or he does not.
6. Record results:

Person	Opened safe first attempt	Read item	Issues observed
Kevin
Greg

Any failure (wrong combination, forgotten procedure, lock jam) is a finding. Re-train and re-verify before closing.

---

3. Signal OOB Comms Group Test

The Signal group is the out-of-band channel when Entra / email / Teams may be compromised. See also runbooks/out-of-band-comms-plan.md.

1. From Rory's phone, send a test message to the Signal group: BGKIT annual verification [YYYY-MM-DD] — acknowledge with name.
2. Wait up to 30 minutes. Kevin and Greg must each reply with their name.
3. Record who replied and how long they took.
4. If either fails to reply:
   • Is Signal installed on their personal phone? (Required — not the work phone which may be compromised during an incident.)
   • Is their number still correct in the group?
   • Are notifications enabled?
   • Fix and re-test within one week.

Group member	Replied	Response time	Issues
Rory (sender)	N/A	N/A
Kevin
Greg

The group membership must be exactly Rory, Kevin, Greg — remove anyone who has left the organisation.

---

4. Offline Backup Readability Check

Per runbooks/offline-backup-procedure.md the kit includes an encrypted USB with Terraform state backup and critical runbook exports, plus printed credential list and key contacts.

4a. USB readability

1. Take the encrypted USB from the safe.
2. Insert it into a clean, trusted workstation (the same admin workstation used for step 1d, not a user device).
3. Unlock the encryption using the passphrase (stored separately — printed in a sealed envelope in the same safe, split from the USB).
4. Mount the volume and list the contents. Expected contents:
   • terraform-state/ — the quarterly state snapshot, per repo
   • runbooks/ — exported copies of the top 10 runbooks (incident-response, DR failover, common-tasks, etc.)
   • contacts.txt — printed key contact list (redundant with the printed copy)
   • README-recovery.md — how to use the backup if Azure and GitHub are both gone
5. Check the modified date of each top-level folder. None should be older than one year. If any is older than 12 months, the offline backup procedure has not been followed — raise a finding.
6. Open one file per folder to confirm it is readable (not corrupted). Prefer terraform-state/azure-infra.tfstate.backup and runbooks/incident-response.md.

4b. Printed credential list freshness

1. Open the sealed printed credential envelope.
2. For each account in the kit scope list, confirm the printed entry includes:
   • Username (full UPN for Entra, root email for AWS, etc.)
   • Current password
   • MFA method and where the token lives
   • Rotation date (month and year)
3. Compare printed rotation dates against the actual rotation log (palo-alto-backup-credential-rotation.md and similar). Any entry more than 12 months out of date is a finding.
4. Compare printed usernames against the current live list. Any account present in production but missing from the printed kit is a finding. Any account in the printed kit that no longer exists is also a finding (remove it).

Record:

Item	Readable	Last refresh date	Issues
USB mount
terraform-state snapshot
runbook exports
Printed credential list
Printed contact list

Re-seal any envelopes opened during verification.

---

5. Update Printed Credentials if Changed

If any break-glass credential has rotated since the last verification — or if the verification itself revealed a stale printed copy — update the printed kit before returning it to the safe.

Procedure

1. Print the new credential on a single dedicated printer (not the shared MFP). The printer must not retain a spooled copy or log job history — wipe the print queue and local cache after the job.
2. Use a tamper-evident envelope. Write the rotation date on the outside in ink.
3. Place the new sheet in the envelope, seal it, sign across the seal.
4. Shred the old sheet immediately — cross-cut shredder, in Rory's physical presence. Do not take it away to shred later.
5. Place the sealed new envelope back in the safe.
6. Update the printed credential rotation log (inside the safe) with date, account, and who performed the rotation.
7. If the hardware MFA token was also replaced, the new token joins the safe and the old token is destroyed (hardware shredder or vendor return) — never re-used for a non-break-glass account.

What NOT to do

• Do not photograph the printed sheet to "keep a digital copy." The whole point is that no digital copy exists.
• Do not email the credential to yourself "temporarily." Ever.
• Do not keep the shredded bag onsite — dispose of it in an external secure bin the same day.

---

6. Closeout

After all sections are complete:

1. Record overall result: PASS (all sections green) or FAIL (any finding).
2. File the completed verification log in bedrock-docs/compliance/annual-review-checklist.md (link to dated log) and in the SharePoint compliance evidence folder Adriana maintains.
3. Register evidence in SecOps via POST /api/evidence with control_id = "BGKIT-ANNUAL", title = "Annual break-glass kit verification YYYY", and the completion date in period_end.
4. Open a follow-up story in SecOps for every finding, linked to this verification. Do not close the verification while findings are open — mark it as PASS-with-followups.
5. Schedule next year's verification on the calendar now. Do not wait.

---

Document History

Date	Change	Author
April 2026	BGKIT-004: Initial annual verification runbook — printed login, safe access, Signal, offline backup, credential refresh	Kobe

Related Documentation

• Out-of-Band Communications Plan
• Offline Backup Procedure
• Crown Jewel Inventory
• Incident Response
• Palo Alto Backup Credential Rotation