Docs

Appearance

Maester M365 Security Audit Guide ​

Purpose ​

This document explains how the Maester M365 security audit works, how to read and act on results, how to remediate failing tests, and how to document intentional deviations.

Maester is an open-source test framework that runs 280+ automated tests against Microsoft 365 and Entra ID configuration, checking against CISA SCuBA baselines, EIDSCA (Entra ID Security Config Analyzer), and Microsoft ORCA recommendations. It runs weekly in both Production (ciriusgroup.com) and DDE (ciriusdde.com) tenants.

Workflow Schedule and Trigger ​

TriggerTimeScope
Scheduled (cron)Monday 14:00 UTC (7:00 AM PT / 10:00 AM ET)Both tenants in parallel
Manual (workflow_dispatch)On demandBoth tenants in parallel

Both tenants run simultaneously (fail-fast: false). A failure in one tenant's job does not block the other.

Workflow: ops-automation/.github/workflows/maester-m365-audit.yml

What Maester Tests ​

Maester runs tests organized by category and tenant scope:

EIDSCA — Entra ID Security Config Analyzer ​

Tests Entra ID (Azure AD) configuration settings against security baselines:

  • EIDSCA.AP — Authorization policy (who can register apps, create tenants, create groups, invite guests, join devices)
  • EIDSCA.AG — Group settings (guest access to M365 groups, group creation restrictions)
  • EIDSCA.PR — Authentication methods (FIDO2, Authenticator, OATH TOTP configuration)
  • EIDSCA.ST — Security and tenant settings

CISA SCuBA — Secure Cloud Business Applications ​

Tests against the CISA Microsoft 365 Secure Configuration Baseline:

  • CISA.MS.AAD.2.x — MFA enforcement, phishing-resistant MFA, Conditional Access for MFA
  • CISA.MS.AAD.3.x — Conditional Access: device compliance, risk-based, legacy auth blocking
  • CISA.MS.AAD.5.x — Non-admin controls: app registration, tenant/group creation restrictions
  • CISA.MS.AAD.7.x — Privileged access: PIM, permanent role assignments, emergency access
  • CISA.MS.AAD.8.x — Guest user controls

Tests Exchange Online and Defender for Office 365 settings:

  • Anti-phishing policies (impersonation protection, mailbox intelligence)
  • Anti-spam policies (inbound / outbound)
  • Anti-malware policies
  • Safe Attachments and Safe Links policies

Maester Custom Tests (MT.xxxx) ​

Maester-specific tests for settings not covered by CISA/EIDSCA:

  • MT.1001–MT.1036 — Conditional Access policy gaps
  • MT.1055–MT.1070 — Authorization policy details (app registration, group creation, device join)
  • MT.1090 — Device registration (global admin at Entra join)
  • MT.1100–MT.1103 — Intune diagnostic settings, Feature Update policy, Restricted Admin Units

Exchange Online Tests ​

Exchange Online and Security & Compliance tests run when the include_exchange: true flag is set. These require the kobe-audit service principal to be registered in Exchange Online (see Exchange Setup below).

  • Accepted domain authentication policies
  • DMARC/DKIM/SPF enforcement (flagged as DNS category)
  • Transport rules and connectors

Teams and SharePoint Tests ​

Basic Teams and SharePoint security configuration checks run when include_teams: true:

  • External access policies
  • Guest sharing settings
  • Meeting policies

Purview / Compliance Tests ​

Requires IPPS (Security & Compliance) connection — runs alongside Exchange tests. Checks:

  • Sensitivity label configuration
  • Retention policy coverage
  • Unified audit log status

How to Read Results ​

Email Report Format ​

The weekly email shows:

Production    DDE
92.3%         87.1%
180 passed    169 passed
15 failed     25 failed
42 skipped    38 skipped

Score interpretation:

  • ≥ 90% — Good. Investigate failed tests but no urgent action.
  • 70–89% — Fair. Prioritize remediation of Terraform and PowerShell fixes.
  • < 70% — Poor. Review all failed buckets; schedule remediation work.

Effective Pass Rate Calculation ​

pass_rate = passed / (passed + failed) × 100

Skipped tests are excluded from the score. A skipped test means a prerequisite was not met (e.g., a license feature is not available, or the test requires a resource that doesn't exist). Skipped tests are shown in the email for transparency but do not count against the score.

Remediation Buckets ​

Each failing test is automatically categorized into one of five remediation tracks:

BucketMeaningWho Acts
terraformFix by adding or updating a Terraform resource in azure-infra or azure-dde-infraKobe — create PR
powershellFix by running a Graph PowerShell or Exchange Online PowerShell commandKobe — run command and verify
dnsFix by updating a DNS record (SPF, DKIM, DMARC)Rory — update DNS zone
manualRequires a portal action or licensing decisionRory — decide and action
reviewNo rule matched; needs human triage to determine which track appliesKobe investigates, Rory decides

The email groups failing tests by bucket. Work top-to-bottom: terraform → powershell → dns → manual → review.

Understanding SKIP Results ​

A SKIP is not a failure. Common skip reasons:

  • Feature requires Entra ID P2 (not available in all configurations)
  • Test requires a specific policy or resource to exist (nothing to test against)
  • Exchange Online connection failed (see Exchange Setup)
  • Test not applicable to this tenant configuration

If a test that was previously passing changes to SKIP, investigate — it may indicate a configuration was removed.

Remediation Tracks ​

Terraform Track ​

Terraform-fixable failures correspond to Entra ID resource configuration managed in azure-infra/entra/ and azure-dde-infra/entra/. Common resources:

hcl
# Authorization policy (app registration, tenant creation, group creation, guest invites)
resource "azuread_authorization_policy" "main" { ... }

# Group settings (guest access to M365 groups, group creation restrictions)
resource "azuread_group_settings" "main" { ... }

# Conditional Access policies
resource "azuread_conditional_access_policy" "mfa_all_users" { ... }

# Administrative unit with restricted membership management
resource "azuread_administrative_unit" "privileged" {
  is_member_management_restricted = true
}

# Intune diagnostic settings
resource "azurerm_monitor_diagnostic_setting" "intune" { ... }

Workflow: create a branch, update the relevant Terraform resource, open a PR. CI runs plan and Checkov. After Rory's review and merge, verify Maester passes next run or on-demand run.

Do not use ACCEPTED_RISK status. If a setting cannot be remediated, document it as an intentional deviation (see below).

PowerShell Track ​

PowerShell-fixable failures require running Graph PowerShell or Exchange Online PowerShell commands as the kobe-audit service principal or as a Global Admin:

powershell
# Example: Disable global admin at Entra device join
Update-MgPolicyDeviceRegistrationPolicy -LocalAdminPassword @{ IsEnabled = $true } `
    -AzureADJoin @{ IsAdminConfigurable = $false }

# Example: Update authentication method policy
Update-MgPolicyAuthenticationMethodPolicy -IsRegistrationCampaignEnabled $true

# Example: Update security defaults
Update-MgPolicyIdentitySecurityDefaultEnforcementPolicy -IsEnabled $false

Run commands from a session authenticated as Global Admin. Verify in Maester next run.

DNS Track ​

DNS-category failures require updating DNS records for the domain:

TestRecordAction
DMARC not configured_dmarc.ciriusgroup.com TXTAdd/update DMARC record
DKIM not configuredSelector recordsEnable DKIM in Exchange Online (auto-creates DNS records)
SPF not strict enoughciriusgroup.com TXTTighten SPF record

DNS changes are made in the DNS provider (Azure DNS or the external registrar). Allow up to 48 hours for propagation before re-testing.

Manual Track ​

Manual-track failures require a portal decision or licensing action:

  • License-gated features (e.g., Entra ID P2 for PIM activation approval, Identity Protection risk policies)
  • Features that require manual portal configuration not exposed via Graph API
  • Policy decisions that require Rory's explicit sign-off

Document the outcome in the Intentional Deviations section of this guide or as a comment in the relevant Terraform resource.

Review Track ​

When categorization rules don't match a failing test:

  1. Look up the test in the Maester documentation or EIDSCA reference
  2. Identify whether the fix is Terraform, PowerShell, DNS, or manual
  3. Rory decides on remediation priority and track
  4. Update scripts/categorize-maester.py RULES list to auto-classify the test in future runs

How to Classify a New Failing Test ​

When a previously passing test starts failing, or a new test appears in the failed list without a category rule:

  1. Note the test name and tag from the email report
  2. Search scripts/categorize-maester.py RULES for any existing pattern matching the tag or test name
  3. If no match exists, add a rule to the RULES list:
    python
    (r"EIDSCA\.XX|relevant-keyword",
 "terraform",  # or powershell / dns / manual
 "Human-readable note about what needs to change"),
  4. Open a PR for the rule addition — this keeps future categorization automatic
  5. Simultaneously address the underlying failure in the appropriate track

How to Document Intentional Deviations ​

Some tests may fail because a security control is intentionally not applied due to:

  • Operational constraints (e.g., legacy application compatibility)
  • Risk accepted after review
  • Control implemented by compensating means

Document intentional deviations here:

Current Intentional Deviations ​

TestTenantReasonCompensating ControlReviewed ByReview Date
(none at this time)

When adding a deviation:

  1. Add a row to the table above
  2. Add a comment in the relevant Terraform resource (if applicable):
    hcl
    # MAESTER-DEVIATION: <Test ID> — <reason>. Reviewed by Rory on <date>.
# Compensating control: <description>
  3. Do not suppress findings in SecOps — Maester findings are automatically posted and should remain visible

How to Run Maester On Demand ​

Via GitHub Actions UI ​

  1. Go to ops-automation → Actions → Maester M365 Security Audit
  2. Click Run workflow → Run workflow (uses default branch: main)
  3. Both Production and DDE run in parallel

Via GitHub CLI ​

bash
gh workflow run maester-m365-audit.yml --repo Cirius-Group-Inc/ops-automation

Viewing Results ​

  1. Go to the workflow run in GitHub Actions
  2. Expand the Maester - Production and Maester - DDE job outputs
  3. Download the artifact: maester-Production-<timestamp> or maester-DDE-<timestamp>
  4. The artifact contains:
    • test-results.json — full test result data
    • test-results.html — Maester HTML report (open in browser for detailed test view)
    • maester-summary.json — summary used by email builder

Artifacts are retained for 90 days.

Exchange Setup Requirement ​

Exchange Online tests require the kobe-audit service principal to be registered in Exchange Online with the Exchange.ManageAsApp API permission (Terraform-managed in azure-infra/entra/maester-graph-permissions.tf and azure-dde-infra/entra/maester-graph-permissions.tf).

Additionally, for ORCA anti-spam/anti-phishing tests, the service principal needs the View-Only Organization Management role in Exchange Online. This role is NOT set via Graph API — it requires manual registration as an Exchange service principal:

powershell
# Run as Global Admin in Exchange Online PowerShell
Connect-ExchangeOnline -UserPrincipalName admin@ciriusgroup.com

# Register the service principal in Exchange Online
New-ServicePrincipal `
    -AppId <MAESTER_CLIENT_ID> `
    -ObjectId <SP_OBJECT_ID> `
    -DisplayName "kobe-security-audit"

# Grant View-Only Organization Management role
Add-RoleGroupMember "View-Only Organization Management" -Member "kobe-security-audit"

If Exchange Online is not pre-connected, EXO-dependent tests are skipped (not failed). The workflow logs a warning and continues — all Graph API and Entra ID tests still run and score.

Current status per tenant:

TenantExchange ConnectionNotes
Production✅ Configuredgithub-actions-maester app registration
DDE⚠️ Partialkobe-audit — verify View-Only Configuration role is assigned

Compliance Evidence Mapping ​

Maester results serve as automated audit evidence for the following controls:

HIPAA ControlRequirementMaester Coverage
§164.312(d) — Person or entity authenticationMFA enforcedCISA.MS.AAD.2.x — MFA enforcement tests
§164.312(a)(1) — Unique user identificationAuthentication methods configured correctlyEIDSCA.PR authentication method tests
§164.308(a)(5)(ii)(D) — Password managementStrong authentication enforcedEIDSCA.AP, Authenticator phishing-resistant tests
§164.308(a)(3)(i) — Workforce access managementPrivileged access controlledCISA.MS.AAD.7.x PIM and permanent role tests
§164.312(b) — Audit controlsAudit logging activeMT tests for unified audit log
§164.308(a)(1)(ii)(C) — Information system activity reviewRegular configuration reviewWeekly Maester run provides automated evidence
SOC2 ControlHow Maester Covers It
CC6.1 — Logical access securityCA policies, MFA enforcement tests
CC6.2 — AuthenticationAuthentication method and phishing-resistant MFA tests
CC6.3 — Access removalGuest access, stale identity tests
CC6.6 — Logical access controlsConditional Access, authorization policy tests
CC7.2 — System monitoringWeekly run provides continuous configuration monitoring
CC9.1 — Risk assessmentORCA anti-phishing and email security tests

Maester results are automatically posted to SecOps via scripts/post_findings_secops.py after each run. These findings are visible in the SecOps dashboard and contribute to the compliance evidence trail.

Troubleshooting ​

SymptomCauseFix
All tests skipped for one tenantOIDC auth failed — federated credential mismatchCheck kobe-audit or github-actions-maester federated credential subject claim
EXO tests all skippedExchange Online not pre-connected or View-Only Configuration role missingSee Exchange Setup
IPPS/Purview tests skippedSecurity & Compliance connection failedVerify IPPS token acquisition in workflow logs; check kobe-audit RBAC
Pass rate dropped significantly week-over-weekPolicy change, tenant change, or new test addedReview what changed in the failed tests list; check Entra ID audit log for recent changes
artifact_upload collision in matrix runmaester-action's fixed artifact name collidesAlready handled — artifact_upload: false in maester-action, custom upload step below

Document History ​

DateChangeAuthor
April 2026Initial guide — covers test categories, pass rate, remediation tracks, Exchange setup, on-demand run, compliance evidence mappingKobe

Internal use only — Cirius Group