Palo Alto Backup Credential Rotation Runbook

Applies to: palo-alto-backup-kv (Azure Key Vault) and AWS Secrets Manager Rotation schedule: Annually (all secrets expire 2027-04-01; set next expiry when rotating) Who runs this: Rory (sole IT/security administrator)

---

Secrets Covered

Azure Key Vault (palo-alto-backup-kv)

Secret Name	Purpose	Rotation Owner
palo-alto-backup-github-token	GitHub PAT for config push to backup repo	GitHub (generate new PAT)
palo-alto-backup-ses-smtp	AWS SES SMTP credentials for alert emails	AWS IAM
palo-alto-backup-azure-prod-panorama	Panorama API key	Palo Alto Panorama
palo-alto-backup-azure-prod-firewall	Firewall API key	Palo Alto Firewall

AWS Secrets Manager (Networking account — 238342914131)

Secret Name	Purpose	Rotation Owner
palo-alto-backup/github-token	GitHub PAT for config push (AWS DR backup)	GitHub
palo-alto-backup/aws-panorama	Panorama API key (AWS environment)	Palo Alto Panorama
palo-alto-backup/aws-dr-firewall	DR firewall API key	Palo Alto Firewall

---

Pre-Rotation Checklist

• [ ] Confirm the backup container job (palo-alto-backup-aci) is not mid-run (check Azure portal → Container Instances → palo-alto-backup-aci)
• [ ] Note current secret expiry dates before rotating (az keyvault secret list --vault-name palo-alto-backup-kv --query "[].{name:name, expires:attributes.expires}" -o table)
• [ ] Have access to Panorama web UI and GitHub to generate new credentials
• [ ] Identify the next expiry date (one year from today) before starting

---

Rotation Procedures

1. GitHub PAT (palo-alto-backup-github-token / palo-alto-backup/github-token)

Generate new token:

1. GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens
2. Create new token scoped to the backup target repo with Contents: Read and Write
3. Set expiry to 13 months (gives a buffer before the KV expiry alert fires)
4. Copy the token value — it is shown only once

Update Azure KV:

az keyvault secret set \
  --vault-name palo-alto-backup-kv \
  --name palo-alto-backup-github-token \
  --value '{"token":"<NEW_TOKEN>","repo":"<OWNER/REPO>","branch":"main"}'

Update AWS Secrets Manager:

aws secretsmanager put-secret-value \
  --secret-id palo-alto-backup/github-token \
  --secret-string '{"token":"<NEW_TOKEN>","repo":"<OWNER/REPO>","branch":"main"}' \
  --region us-east-1 \
  --profile networking

Revoke old token: Delete the old PAT from GitHub immediately after confirming the new one works.

---

2. SES SMTP Credentials (palo-alto-backup-ses-smtp)

SES SMTP credentials are derived from an IAM user's access key.

Generate new SMTP credentials:

1. AWS Console → IAM → Users → palo-alto-backup-ses-user (or equivalent)
2. Security credentials → Create access key
3. Convert access key to SMTP password using the AWS formula:

# AWS SMTP password derivation (Python)
python3 -c "
import hmac, hashlib, base64
DATE = '11111111'
SERVICE = 'ses'
MESSAGE = 'SendRawEmail'
TERMINAL = 'aws4_request'
REGION = 'us-east-1'
secret_key = '<NEW_SECRET_KEY>'
kDate    = hmac.new(('AWS4' + secret_key).encode(), DATE.encode(), hashlib.sha256).digest()
kRegion  = hmac.new(kDate, REGION.encode(), hashlib.sha256).digest()
kService = hmac.new(kRegion, SERVICE.encode(), hashlib.sha256).digest()
kTerm    = hmac.new(kService, TERMINAL.encode(), hashlib.sha256).digest()
final    = hmac.new(kTerm, MESSAGE.encode(), hashlib.sha256).digest()
print(base64.b64encode(b'\x02' + final).decode())
"

The SMTP username is the IAM access key ID. The SMTP password is the value printed above.

Update Azure KV:

az keyvault secret set \
  --vault-name palo-alto-backup-kv \
  --name palo-alto-backup-ses-smtp \
  --value '{"username":"<ACCESS_KEY_ID>","password":"<SMTP_PASSWORD>"}'

Deactivate old IAM access key in the console after confirming email delivery works.

---

3. Panorama API Key

Generate new API key:

curl -k -X GET 'https://<PANORAMA_IP>/api/?type=keygen&user=<ADMIN_USER>&password=<ADMIN_PASS>'

Extract <key> from the XML response.

Update Azure KV (production Panorama):

az keyvault secret set \
  --vault-name palo-alto-backup-kv \
  --name palo-alto-backup-azure-prod-panorama \
  --value '{"ip":"<PANORAMA_IP>","api_key":"<NEW_KEY>"}'

Update AWS Secrets Manager (DR Panorama, if separate):

aws secretsmanager put-secret-value \
  --secret-id palo-alto-backup/aws-panorama \
  --secret-string '{"ip":"<PANORAMA_IP>","api_key":"<NEW_KEY>"}' \
  --region us-east-1 \
  --profile networking

---

4. Firewall API Key

Generate new API key (repeat for each managed firewall):

curl -k -X GET 'https://<FIREWALL_IP>/api/?type=keygen&user=<ADMIN_USER>&password=<ADMIN_PASS>'

Update Azure KV:

az keyvault secret set \
  --vault-name palo-alto-backup-kv \
  --name palo-alto-backup-azure-prod-firewall \
  --value '{"ip":"<FIREWALL_IP>","api_key":"<NEW_KEY>"}'

Update AWS Secrets Manager (DR firewall):

aws secretsmanager put-secret-value \
  --secret-id palo-alto-backup/aws-dr-firewall \
  --secret-string '{"ip":"<FIREWALL_IP>","api_key":"<NEW_KEY>"}' \
  --region us-east-1 \
  --profile networking

---

Update Terraform Expiry Dates

After rotating, update the expiry_date values in Terraform so the next expiry cycle is tracked. This is a separate Terraform PR — do not combine with other changes.

File: azure-infra/modules/palo-alto-backup-azure/main.tf

Update all four azurerm_key_vault_secret resources:

expiry_date = "YYYY-04-01T00:00:00Z"  # Set to one year from rotation date

The lifecycle { ignore_changes = [value] } block means Terraform will not overwrite the rotated secret values — only the expiry date metadata is updated.

Open a PR against azure-infra with the expiry date change. CI runs terraform plan; the plan should show only the expiry date attribute changing on each secret.

---

Post-Rotation Verification

Trigger an on-demand backup

Run the on-demand backup workflow to confirm the new credentials work end-to-end:

GitHub → ops-automation → Actions → "Palo Alto On-Demand Backup" → Run workflow

The workflow stops and restarts the ACI container group, which causes the backup function to run immediately. Check the container logs in Azure portal → Container Instances → palo-alto-backup-aci → Containers → Logs.

Success indicators:

• Container exits with code 0
• A new blob appears in the backup storage container under backups/ within 5 minutes
• No authentication errors in the logs

Confirm expiry monitoring

Run the HIPAA audit manually to confirm the KV expiry section shows healthy status:

GitHub → ops-automation → Actions → "Weekly HIPAA Compliance Audit" → Run workflow

The backup-results.json artifact should show kv_secret_expiry with status: "OK" for all secrets and days_left > 90.

---

Rollback

If the backup function fails after rotation:

1. The ACI container reads secrets from KV at runtime — rollback is immediate.
2. Restore the previous secret value using az keyvault secret set with the old value.
3. Old PAT/API key values should be kept in a local secure note until post-rotation verification passes.
4. Re-run the on-demand backup workflow to confirm rollback worked.

---

Alert Thresholds

The weekly HIPAA audit (backup-audit.sh) monitors expiry dates and raises issues at:

Days Until Expiry	Severity	HIPAA Email Section
≤ 30 days	CRITICAL	Backup issues table
≤ 60 days	HIGH	Backup issues table
≤ 90 days	WARNING	Backup issues table
No expiry date	CRITICAL	Backup issues table

The KV Secret Expiry table in the HIPAA email shows all secrets and their days remaining regardless of status.

---

Notes on AWS Secrets Manager Rotation

AWS Secrets Manager does not have a native expiry date field (unlike Azure Key Vault). Rotation for AWS secrets is tracked by convention — set a rotation-due tag on each secret after rotating:

aws secretsmanager tag-resource \
  --secret-id palo-alto-backup/github-token \
  --tags '[{"Key":"rotation-due","Value":"YYYY-04-01"}]' \
  --region us-east-1 \
  --profile networking

A future enhancement could add an AWS Secrets Manager expiry check to backup-audit.sh by reading this tag.