Docs

Appearance

Runbook: Common Infrastructure Tasks ​

Purpose ​

This runbook provides step-by-step instructions for common infrastructure tasks. It is intended for the IT administrator managing the environment and serves as a reference to ensure tasks are performed consistently and completely.

1. Deploying Infrastructure Changes ​

All infrastructure changes are managed through Terraform and deployed via the GitHub Actions CI/CD pipeline. No manual changes should be made directly in the Azure or AWS portals except in emergencies.

Standard Deployment Process ​

  1. Create a feature branch in the appropriate repository (azure-infra, aws-infra, or azure-dde-infra)
bash
git checkout -b feature/description-of-change

  1. Make changes to Terraform files locally in VS Code

  2. Validate locally before pushing

bash
terraform fmt
terraform validate
terraform plan

  1. Review the plan output — confirm only expected resources are changing

  2. Push the branch and open a pull request on GitHub

  3. Wait for the PR pipeline to complete — security scans must pass:

    • Terraform fmt/validate ✅
    • Checkov HIPAA scan ✅
    • Trivy scan ✅

  4. Merge the PR — Terraform plan and apply run automatically after merge

  5. Review the plan/apply output in the GitHub Actions workflow logs

  6. Verify the deployment in the Azure or AWS portal

Emergency Changes ​

If an emergency change must be made directly in the portal (not via Terraform), document it immediately and create a Terraform PR to bring the code back in sync with the live infrastructure as soon as possible. Undocumented manual changes cause Terraform state drift.

2. Adding a New User ​

New users are provisioned through Entra ID. All accounts require MFA before accessing any system.

Steps ​

  1. Log into the Azure portal → Entra ID → Users → New user

  2. Fill in user details

    • Display name
    • Username (follow existing naming convention)
    • Department and job title

  3. Assign groups and roles based on the user's job function — apply least privilege

  4. Set a temporary password and note it for delivery to the user

  5. Notify the user of their credentials and require them to change their password on first login

  6. Confirm MFA enrollment before granting access to any production systems — do not grant access until MFA is active

  7. For AVD/DDE access — add the user to the appropriate AVD application group in the DDE tenant

  8. Document the access granted — note what systems the user has access to and why

Offboarding a User ​

  1. Disable the Entra ID account immediately upon termination notification
  2. Remove the user from all groups and application assignments
  3. Revoke any active sessions (Entra ID → User → Revoke sessions)
  4. Transfer or archive any data owned by the user
  5. Delete the account after 30 days if no access recovery is needed

3. Firewall Rule Changes ​

Note: As of March 2026, Panorama is being rebuilt. The AWS and DDE Panorama instances failed due to disk space exhaustion and are pending rebuild. The Azure production firewalls have never had Panorama. Until Panorama rebuild is complete, use the Local Firewall Management steps below for all changes.

Firewall changes are made via Panorama for changes affecting multiple environments, or directly on the local firewall management interface for environment-specific changes.

Before Making a Change ​

  • Identify the source zone, destination zone, application, and required action
  • Confirm the change is necessary — deny by default is the correct posture
  • For changes affecting customer-facing traffic (DDE untrust zone) be extra careful — test impact before applying

Steps — Panorama (Multi-Environment) ​

  1. Log into Panorama
  2. Navigate to the appropriate Device Group
  3. Add or modify the security policy rule
  4. Review the rule — confirm source, destination, application, and action are correct
  5. Commit to Panorama
  6. Push to managed devices — select the target firewalls
  7. Verify traffic is flowing as expected after the change

Steps — Local Firewall Management ​

  1. Log into the firewall management interface via the management zone
  2. Navigate to Policies → Security
  3. Add or modify the rule
  4. Commit the change
  5. Verify traffic is flowing as expected

After a Change ​

  • Document what was changed and why in a comment or change log
  • Monitor firewall logs for unexpected traffic patterns after the change

4. Restarting Services ​

Restarting a Windows Service on an Azure VM ​

  1. Connect to the VM via RDP through the management zone — do not expose RDP directly to the internet

  2. Open Services (services.msc) or use PowerShell:

powershell
# Restart a specific service
Restart-Service -Name "ServiceName" -Force

# Check service status
Get-Service -Name "ServiceName"

  1. Verify the service is running after restart

  2. Check application logs to confirm the application is functioning correctly after the restart

Restarting MySQL ​

powershell
# Restart MySQL service
Restart-Service -Name "MySQL80" -Force

# Verify MySQL is running
Get-Service -Name "MySQL80"

# Test connectivity
mysql -u admin -p -e "SELECT 1;"

Restarting the Palo Alto Firewall ​

⚠️ Restarting the firewall will drop all active connections. Notify users before proceeding.

  1. Log into the firewall management interface
  2. Navigate to Device → Setup → Operations
  3. Click Reboot Device
  4. Wait for the firewall to come back online (typically 3-5 minutes)
  5. Verify all interfaces are up and traffic is flowing

5. Onboarding a New Server ​

New servers are built from VM images created in the dev environment and deployed via Terraform.

Steps ​

  1. Build the VM image in dev using the standardized base image process

    • Apply OS hardening
    • Install required software
    • Configure required services
    • Run Trivy scan against the image before promoting

  2. Write or update Terraform in the appropriate repository to define the new VM

    • Use existing VM module patterns for consistency
    • Place the VM in the correct subnet and security zone
    • Apply standard tags (environment, owner, purpose, cost center)

  3. Open a PR and let the pipeline validate and plan the change

  4. After deployment confirm the following:

    • [ ] VM is reachable from expected sources only
    • [ ] VM is NOT reachable from unexpected sources (test from outside the allowed zone)
    • [ ] Entra ID join successful (if applicable)
    • [ ] Backup configured in Veeam
    • [ ] Monitoring and alerting configured
    • [ ] NSG rules reviewed and tightened to minimum required access
    • [ ] VM appears in vulnerability scanning scope

  5. Update documentation — add the new server to relevant architecture docs

6. Rotating Credentials and Secrets ​

Credentials should be rotated on a regular schedule and immediately if a potential compromise is suspected.

GitHub Actions OIDC (No Rotation Required) ​

GitHub Actions uses OIDC federated credentials — there are no long-lived secrets to rotate. If the OIDC trust relationship needs to be updated, modify the federated credential in Entra ID or the AWS IAM role trust policy.

Azure Key Vault Secrets ​

  1. Log into the Azure portal → Key Vault → Secrets
  2. Select the secret to rotate
  3. Click New Version and enter the new value
  4. Update any applications or services referencing the secret to use the new version
  5. Verify applications are functioning with the new secret
  6. Disable the old version after confirming the new version is working

Entra ID Service Principal Credentials ​

  1. Log into Entra ID → App Registrations → select the app
  2. Navigate to Certificates & Secrets
  3. Add a new client secret with an appropriate expiry
  4. Update any services using the old secret to use the new one
  5. Verify services are functioning
  6. Delete the old secret

MySQL Passwords ​

sql
-- Rotate a MySQL user password
ALTER USER 'username'@'host' IDENTIFIED BY 'new_password';
FLUSH PRIVILEGES;

Update any application connection strings referencing the old password after rotating.

Emergency Credential Rotation (Suspected Compromise) ​

If a credential compromise is suspected:

  1. Rotate immediately — do not wait to confirm the compromise
  2. Revoke the old credential as soon as the new one is in place
  3. Review audit logs for unauthorized use of the compromised credential
  4. Follow the incident response runbook if unauthorized access is confirmed

See Incident Response Runbook for full breach response steps.

Document History ​

DateChangeAuthor
March 2, 2026Corrected pipeline deployment process to reflect actual PR vs post-merge stagesRory
March 2026Added Panorama rebuild status note to firewall sectionRory
February 2026Initial draftRory

Internal use only — Cirius Group