Key Learnings — Infrastructure Gotchas and Architecture Principles

Consolidated reference of hard-won lessons from operating Cirius Group infrastructure. Updated continuously — add entries as new issues are discovered.

Last reviewed: 2026-04-16

---

Azure / Entra / ARM

SecurityEvent Lives in the Sentinel-Enabled LAW Only

Microsoft routes SecurityEvent (EventID 4688, 4624, 4662, etc.) only to the Log Analytics workspace where Microsoft Sentinel is enabled — not to secondary or ops workspaces. Querying OPS_LAW_WORKSPACE_ID for SecurityEvent returns empty silently.

• PROD: SecurityEvent → LAW_WORKSPACE_ID (5d76d1f2, cirius-logging-law-central)
• DDE: SecurityEvent → DDE_LAW_WORKSPACE_ID (2176be20, ciriusdde-logging-law-central)
• Ops workspaces (OPS_LAW_WORKSPACE_ID, DDE_OPS_LAW_WORKSPACE_ID) contain Event table and syslog only.

Lesson: Always verify you're querying the correct workspace before assuming a KQL query has no results.

Azure VM Names Are Immutable

A VM name is baked into the resource ID and cannot be renamed in place. Renaming requires destroy + recreate. Treat every VM name decision as permanent.

Contrast: AWS resources use the Name tag and can be renamed with zero downtime.

azuread Provider 3.x Removed Several Resources

azuread 3.0 removed azuread_authorization_policy and other high-level policy resources. These must now be applied via null_resource + az rest Graph API calls, triggered by a sha256(config_hash) trigger so they only re-apply on change.

Graph API Race Condition on Consent Grants

When granting new app role permissions (azuread_app_role_assignment) in the same Terraform apply as resources that call Graph API (null_resource provisioners), the provisioners run before consent is fully propagated — resulting in 403 Forbidden.

Fix: Add depends_on = [azuread_app_role_assignment.xxx] to every null_resource that requires that permission. This forces the consent grant to settle before the API call runs.

Palo Alto Marketplace Images: No encryption_at_host

Palo Alto marketplace VM images don't support encryption_at_host = true. Use CMK (Customer-Managed Key) via disk encryption set as the compensating control.

OIDC Federated Credentials: Exact Repository Name Required

OIDC federation for GitHub Actions is case-sensitive and requires the exact org/repo name. Use Cirius-Group-Inc — not CiriusGroup or any other variant. A mismatch causes silent authentication failures.

KQL Hostname Normalization

When joining SecurityEvent/Heartbeat against other tables, normalize hostnames with:

toupper(tostring(split(Computer, ".")[0]))

This handles FQDN vs short hostname mismatches (e.g., ACTDIRAZP01.ciriusgroup.com vs ACTDIRAZP01).

Azure Cost API: End Dates Are Inclusive

Azure Cost Management API treats the timePeriod.to date as inclusive. AWS Cost Explorer treats its end date as exclusive. Always account for this when building cross-cloud cost comparisons.

string.Template Conflicts with CSS Dollar Signs

Python's string.Template uses $ as a substitution marker. HTML/CSS files containing $var will raise KeyError. Use with .replace() instead of string.Template for HTML templates.

Soft Delete + Immutability Default on Recovery Services Vaults

All RSVs should have soft delete enabled and immutability set to Unlocked (not Locked — Locked prevents deletion even for authorized operators). Exception: decommission-bound vaults where immutability blocks cleanup.

MDE Sense Passive Mode Required with Cortex XDR

If both Microsoft Defender for Endpoint and Cortex XDR are active on the same host, MDE Sense must run in Passive Mode only. Active Mode with Cortex XDR causes CPU saturation.

Bash Subshell Variable Loss in While Loops

Piping output into a while read loop runs the loop body in a subshell; any variable modifications are lost when the loop exits. Use process substitution instead:

# WRONG — $count is always 0 after loop
cat file.txt | while read line; do count=$((count + 1)); done

# CORRECT — process substitution keeps variables in current shell
while read line; do count=$((count + 1)); done < <(cat file.txt)

SES Email Delivery with Large MIME Payloads

For HTML emails over ~50KB, use --cli-input-json file://payload.json with base64 encoding. Inline --message arguments fail silently for large payloads.

Checkov: No Dedicated HIPAA Framework Flag

Checkov does not have a --framework hipaa flag. Scan all checks and filter results by HIPAA-relevant check IDs (encryption, logging, network, access). Use Checkov's --check flag with a curated list of IDs.

AWS Security Hub Has No HIPAA Standard

Security Hub's built-in standards do not include HIPAA. Use AWS Audit Manager with the HIPAA Omnibus (January 2013) framework for HIPAA evidence collection. Separately, use Security Hub NIST 800-53 R5 for the compliance posture signal.

ZOMBIE CONTAINER APP — Must Clear Before Re-apply

A failed Container App deployment leaves a shell resource in Azure that Terraform cannot overwrite on retry. The apply will fail on subsequent runs until cleared.

Fix:

1. az containerapp delete --name <name> --resource-group <rg>
2. terraform state rm <resource_address>
3. Then re-run terraform apply

Always check the portal for zombie Container Apps when a deployment fails mid-way. Do not retry the same apply without clearing the shell first.

---

AWS

AWS VM Names Use the Name Tag (Rename is Free)

Unlike Azure, AWS EC2 instances use the Name tag for display — changing it is a non-destructive metadata update with zero downtime.

SES Delivery Region Matters

SES sending must be configured in the region where the SES identity (email/domain) is verified. The SES region may differ from the Lambda region. Always use --region <ses-region> explicitly in aws ses send-email calls.

---

Terraform

Import Before Fixing

Get unmanaged resources into Terraform state with a zero-diff import first. Then remediate in a separate PR. Mixing import + remediation in a single apply often causes unexpected destroy/recreate cycles.

Modules-First Before HIPAA Remediation

Bake HIPAA controls into shared modules before remediating individual resources. Resources become compliant automatically when refactored to use the module. Remediating raw resources creates n separate PRs instead of one module PR.

Never Apply Locally in a HIPAA Environment

All Terraform applies go through GitHub Actions CI/CD after PR merge. Local apply bypasses the CM ticket, the plan review, and the audit trail. It also risks state divergence if CI applies later.

Decommission-Bound Resources: Flat Import, No Hardening

Tag decommission-bound resources with Lifecycle = "decommission" and use a flat import block with a comment marker. Don't invest time hardening resources scheduled for deletion (target: June 2026).

---

Security Monitoring

Gap Detection Pattern for Kill Chain Agents

When a detection agent returns 0 results, it's ambiguous — could be "no attack" or "audit policy not configured." Add a secondary gap-detection query that checks whether the underlying event type (e.g., EventID 4662) has any events in the past 7 days.

• Count > 0: data is flowing, no attack detected. ✓
• Count = 0: audit policy may not be enabled on the source. Emit a warning. ✗

DCSync Detection (MITRE T1003.006)

DCSync is detected via EventID 4662 on domain controllers. Three conditions must match:

1. EventID = 4662
2. Properties field contains at least one DS-Replication GUID:
   • 1131f6aa (DS-Replication-Get-Changes)
   • 1131f6ad (DS-Replication-Get-Changes-All)
   • 89e95b76 (DS-Replication-Get-Changes-In-Filtered-Set)
3. SubjectUserName does NOT end in $ (machine accounts = legitimate DC replication)

These events are in LAW_WORKSPACE_ID (Sentinel workspace), not the ops LAW.

Cortex Hostname Classifier (cortex-audit.py)

The location_label_from_name() function in ops-automation/scripts/cortex-audit.py classifies endpoints into environments by hostname pattern. Known exceptions that required explicit rules:

Hostname	Issue	Fix
ACTRODC01	No AZP/AZT suffix → fell to "Home"	Added ACTRODC → "Prod"
cloudpc	No AWP/AWT → fell to "Home"	Added exact match → "AWS"

When adding a new VM that doesn't follow standard naming, add an explicit rule here.

Break-Glass Accounts Are Permanently Unsuppressible

cirius-breakglass@ciriusgroup.com, cirius-breakglass@ciriusdde.com, and all 7 AWS root accounts must never appear in known-good suppression rules. Any activity from these accounts = CRITICAL incident, no exceptions. This is enforced at the API level in SecOps — even if someone adds a rule, it won't suppress break-glass hits.

---

CI/CD and GitHub Actions

SHA-Pin All Actions References

Every third-party GitHub Action must be pinned to a commit SHA, not a mutable tag. Unpinned tags (@v4, @latest) are a supply-chain attack vector — the action can be updated to malicious code at any time.

# WRONG
uses: actions/checkout@v4

# CORRECT
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2

Container App Deploy: Check for Zombie Before Retrying

See "ZOMBIE CONTAINER APP" in the Azure section above. Never retry a failed Container App apply without first clearing the shell resource.

---

Domain / DNS

Pan_analyzer Has No Reliable pip Package

Don't attempt to pip install pan-python or similar for Palo Alto config analysis. Build custom XML-parsing checks instead using Python's xml.etree.ElementTree.

---

On-Premises / Active Directory

EventID 4662 Must Originate from DCs

EventID 4662 (Directory Service Object Access) is only logged on domain controllers when "Audit Directory Service Access" subcategory is enabled (success). The events flow from DCs → NPS → LAW via the Azure Monitor Agent. Confirm with:

auditpol /get /subcategory:"Directory Service Access"

If the policy is not set, DCSync attacks are invisible in LAW.

---

SecOps Platform

Story API: Start Before Close

The SecOps stories API requires calling /start before /close. Calling /close on a BACKLOG story fails silently (returns 200 but status doesn't change). Always call POST /api/stories/{id}/start first.

Source System Required on Every Incident

Every incident payload must include source_system (PROD, DDE, or AWS). The ioc_key must also include source_system to prevent cross-environment deduplication — two identical events in PROD and DDE must create separate incidents.