DDE Environment Operations

Environment: Azure DDE (tenant ff1c5d68, ciriusdde.com) Compliance scope: HIPAA — Medicare customers, ePHI in scope Infra repo: azure-dde-infraMonitoring job: job-orchestrator-cirius (Container Apps, rg-logging-logs, logging subscription)

1. What DDE Is — Quick Orientation for Someone Who Knows PROD

DDE is a completely separate Azure environment from PROD. There is no shared tenant, no shared subscriptions, no network connectivity, and no shared identity plane. If you know the PROD environment, treat DDE as a different company that happens to be operated by the same person.

Attribute	PROD	DDE
Tenant domain	`ciriusgroup.com`	`ciriusdde.com`
Tenant ID	`d477c9f8`	`ff1c5d68`
Primary purpose	Internal users, infra, SecOps	Customer-facing AVD for Medicare
User access method	Twingate for internal resources	AVD client or browser — no Twingate
Entra ID	PROD Entra	Separate DDE Entra
Entra Connect	Syncs from `ciriusgroup.internal`	Syncs from its own on-prem AD (`ENTRACAZC01`)
Subscriptions	Production, Logging, Identity, Firewall, Dev, Main	Logging, Main, Billings (AVD), Firewall
Firewalls	Palo Alto VM-Series, no Panorama (Azure PROD)	Palo Alto VM-Series + DDE Panorama
SecOps monitoring	`job-orchestrator-cirius`	`job-orchestrator-cirius`
Network to PROD	None — hard isolation	None — hard isolation
Compliance scope	SOC2, internal baseline	HIPAA, HITRUST, SOC2 — Medicare ePHI

What the DDE Firewall subscription hosts:

Palo Alto VM-Series firewall
DDE Panorama (PAN-OS 12.1.2, rebuilt March 2026 — firewall reconnection pending a reboot)
ACTDIRAZC01, ACTDIRAZC02 — DDE Active Directory domain controllers
ENTRACAZC01 — DDE Entra Connect server

What the DDE Billings subscription hosts:

AVD host pool and session hosts (bc-host-*)
Customer VNet (Billings VNet)
Application servers delivering Medicare billing apps

VNet topology: two VNets, peered. Firewall VNet (north/south inspection + AD) peered to Billings VNet (session hosts). All customer internet traffic enters through the Palo Alto firewall zone untrust before reaching AVD session hosts in trust.

How customers connect: Azure Virtual Desktop client application or browser-based HTML5 client, directly over the internet. There is no Twingate in the path — customer access is not via a corporate VPN. The Palo Alto firewall is the enforcement point for customer traffic, not Twingate.

2. AVD Session Host Management

All AVD operations begin at: Azure Portal (DDE tenant) → Azure Virtual Desktop → Host Pools

The DDE host pool name follows the pattern visible in the portal. Session hosts are listed under: Host Pool → Session Hosts

Checking Host Pool Health

Portal:Azure Virtual Desktop → Host Pools → [pool name] → Session Hosts

Each host shows:

Status: Available, Unavailable, Needs Assistance, Shutdown, Upgrading
Sessions: current active session count
Health check status: last health check result (agent, domain join, etc.)

Click any host to see the health check details — each check type (domain join, app health, FSLogix, etc.) reports pass/fail independently.

PowerShell:

powershell

# List all session hosts and their status
Get-AzWvdSessionHost -ResourceGroupName <rg-name> -HostPoolName <pool-name> |
  Select-Object Name, Status, Sessions, LastUpdateTime, AgentVersion |
  Format-Table -AutoSize

CLI:

bash

az desktopvirtualization sessionhost list \
  --resource-group <rg-name> \
  --host-pool-name <pool-name> \
  --output table

Health check failure codes to watch:

Health Check	What it means when it fails
DomainJoinedCheck	Host lost domain join — user logins will fail
DomainTrustCheck	Secure channel to AD is broken
FSLogixHealthCheck	Profile storage unreachable — users may get temp profiles
AppAttachHealthCheck	MSIX app attach failed — published apps may not launch
MonitoringAgentCheck	Azure Monitor Agent not running — telemetry gap
SupportedEncryptionCheck	Encryption config mismatch

Draining a Session Host

Drain mode stops new sessions from routing to a host while existing sessions continue. Use this before maintenance, reboots, or when a host shows health issues.

Portal:Host Pool → Session Hosts → [host] → Properties → Allow new sessions → toggle Off

PowerShell:

powershell

# Enable drain mode (block new sessions)
Update-AzWvdSessionHost `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -Name <host-name> `
  -AllowNewSession:$false

# Confirm drain mode is on
Get-AzWvdSessionHost `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -Name <host-name> |
  Select-Object Name, AllowNewSession, Sessions

Before rebooting a drained host: verify Sessions count is 0 or log off remaining sessions (see Section 3).

Restarting a Session Host

Drain the host (above) and wait for sessions to drop to zero or log them off
Restart via portal or CLI:

Portal:Host Pool → Session Hosts → [host] → Restart

CLI:

bash

# Restart the underlying VM
az vm restart \
  --resource-group <rg-name> \
  --name <vm-name> \
  --subscription <dde-billings-subscription>

PowerShell (after restart, re-enable new sessions):

powershell

Update-AzWvdSessionHost `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -Name <host-name> `
  -AllowNewSession:$true

Wait for the host to return to Available status before re-enabling. Status refresh can take 5–10 minutes after VM boot completes.

Adding Capacity (Scaling the Host Pool)

Temporary capacity — deploy additional session hosts:

In azure-dde-infra, increment the session host count in the Terraform variable for the host pool
Open a PR → CI runs plan → merge → apply
New session hosts appear in the host pool automatically once the VM joins the domain and the AVD agent registers

Immediate manual capacity (break-glass):

If you need a host immediately and cannot wait for a PR cycle:

In the DDE portal: Azure Virtual Desktop → Host Pools → [pool] → Session Hosts → Add session hosts
Select the VM image, size, and subnet (Billings VNet)
Provide domain join credentials for the DDE domain
The host registers to the pool automatically

Document the manual change and follow up with a Terraform PR to bring infra state back into IaC control.

Scaling plan: If a scaling plan is attached to the host pool, review it before adding capacity manually — the plan may be actively draining hosts based on a schedule. Check: Azure Virtual Desktop → Scaling Plans → [plan] → Host Pools

3. User Session Management

Finding Active Sessions

Portal:Azure Virtual Desktop → Host Pools → [pool] → User Sessions

Shows all active sessions: username, session state (Active, Disconnected), session host, client IP, login time.

PowerShell:

powershell

# List all active sessions across the host pool
Get-AzWvdUserSession -ResourceGroupName <rg-name> -HostPoolName <pool-name> |
  Select-Object UserPrincipalName, SessionState, Name, CreateTime |
  Sort-Object CreateTime |
  Format-Table -AutoSize

# Sessions for a specific user
Get-AzWvdUserSession -ResourceGroupName <rg-name> -HostPoolName <pool-name> |
  Where-Object { $_.UserPrincipalName -like "*username*" }

Session states:

Active — user is actively using the session
Disconnected — session exists but user has closed the client (not logged off)
Pending — session is being established

Disconnected sessions consume a session host slot. If a host is full of disconnected sessions and users cannot connect, log off the stale disconnected sessions.

Logging Off Stuck Sessions

Send a message before forcing logoff (HIPAA courtesy — users may have unsaved work):

powershell

# Send a message to a user session before logging them off
Send-AzWvdUserSessionMessage `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -SessionHostName <host-name> `
  -UserSessionId <session-id> `
  -MessageTitle "Session Maintenance" `
  -MessageBody "Your session will be logged off in 5 minutes for maintenance. Save your work."

Force logoff:

powershell

# Get the session ID first
$sessions = Get-AzWvdUserSession `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -SessionHostName <host-name>

# Log off a specific session
Remove-AzWvdUserSession `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -SessionHostName <host-name> `
  -Id <session-id>

Log off all disconnected sessions on a host (batch cleanup):

powershell

$hostName = "<host-name>"
$sessions = Get-AzWvdUserSession `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -SessionHostName $hostName |
  Where-Object { $_.SessionState -eq "Disconnected" }

foreach ($s in $sessions) {
  $id = $s.Name.Split('/')[-1]
  Remove-AzWvdUserSession `
    -ResourceGroupName <rg-name> `
    -HostPoolName <pool-name> `
    -SessionHostName $hostName `
    -Id $id
  Write-Host "Logged off: $($s.UserPrincipalName)"
}

Shadow / Connect to a Session (Admin View)

To connect to an active user session for troubleshooting:

Portal:Host Pool → User Sessions → [session] → Send message (or use the "Connect" option if available for the host pool type)

Via Remote Desktop (direct):

Connect to the session host VM via Bastion or direct RDP (from inside the DDE management network)
On the session host, open Task Manager → Users tab → right-click the user → Connect (requires Local Admin or Remote Desktop Services admin rights on the host)

Note: shadowing a Medicare customer session has HIPAA implications. Document the reason, duration, and scope before connecting to a customer session. Use only for active support incidents, not routine observation.

4. DDE-Specific Admin Tasks

Managing DDE Entra Users vs PROD Entra Users

These are two separate Entra ID tenants. There is no cross-tenant sync, no guest access between them, and no shared user objects.

DDE Entra is reached at:

Portal: https://portal.azure.com → switch to DDE directory (ciriusdde.com)
CLI: az login --tenant ff1c5d68
Entra Admin Center: https://entra.microsoft.com → switch tenant

DDE users are typically:

External Medicare customers onboarded to use AVD-published apps
Break-glass account: cirius-breakglass@ciriusdde.com (exclude from all CA policies)
Service accounts specific to DDE workloads

Creating a DDE user (customer onboarding):

powershell

# Connect to DDE tenant
Connect-MgGraph -TenantId ff1c5d68 -Scopes "User.ReadWrite.All"

$params = @{
  AccountEnabled    = $true
  DisplayName       = "Customer Name"
  MailNickname      = "c.name"
  UserPrincipalName = "c.name@ciriusdde.com"
  PasswordProfile   = @{
    ForceChangePasswordNextSignIn = $true
    Password                      = (New-Guid).ToString().Substring(0,16) + "!"
  }
}
New-MgUser @params

After creation: assign the user to the appropriate AVD Application Group so they can access the published apps: Azure Virtual Desktop → Application Groups → [group] → Assignments → Add users/groups

PROD vs DDE account confusion: A user's @ciriusgroup.com account cannot access DDE resources, and a @ciriusdde.com account cannot access PROD resources. If a user reports they cannot log in to AVD, confirm they are using their DDE credentials, not their internal Cirius credentials.

DDE Conditional Access

DDE has its own CA policies managed in the DDE Entra tenant. They are independent of PROD CA policies — changes to PROD CA do not affect DDE and vice versa.

Viewing DDE CA policies: Portal (DDE tenant) → Entra ID → Security → Conditional Access → Policies

Or via PowerShell:

powershell

Connect-MgGraph -TenantId ff1c5d68 -Scopes "Policy.Read.All"
Get-MgIdentityConditionalAccessPolicy |
  Select-Object DisplayName, State |
  Sort-Object DisplayName |
  Format-Table

DDE CA baseline requirements (HIPAA scope):

Require MFA for all users on all apps — no exceptions except break-glass
Block legacy authentication protocols
Require MFA for Azure Management access
Session sign-in frequency should be set for customer-facing apps (e.g., 8-hour max for AVD sessions handling PHI)

Break-glass exclusion is mandatory: cirius-breakglass@ciriusdde.com must be excluded from all CA policies. Verify before enabling any new policy: CA policy → Users → Exclude → confirm cirius-breakglass@ciriusdde.com is listed

A misconfigured CA policy that locks out break-glass has no self-service recovery.

Testing a CA policy change (always do this first):

Set the new policy to Report-only mode
Sign in as a test DDE user
Review the CA sign-in log to confirm the policy would behave as expected: Entra ID → Sign-in logs → [sign-in] → Conditional Access tab
Switch to On only after confirming report-only results match intent

5. DDE Entra Connect: Sync Health

The DDE Entra Connect server is ENTRACAZC01, located in the Firewall subscription's Firewall VNet. It syncs from the on-prem DDE AD into the DDE Entra tenant.

This is separate from the PROD Entra Connect that syncs ciriusgroup.internal to PROD Entra ID. A failure in DDE sync has no impact on PROD, and vice versa.

Checking Sync Health

On ENTRACAZC01 (connect via Bastion or via Palo Alto jump path through the DDE mgmt zone):

powershell

Import-Module ADSync

# Scheduler status — confirm sync is running on schedule
Get-ADSyncScheduler | Select-Object SyncCycleEnabled, NextSyncCyclePolicyType, NextSyncCycleStartTimeInUTC

# Last sync run result per connector
Get-ADSyncConnectorRunStatus
# Look for Result = "success" and a recent StartDate (within 30 minutes)

Portal (DDE tenant):Entra ID → Entra Connect → Sync status

Last sync time should be within 30 minutes
Object count should match expected user population
Sync errors should be 0 or documented/known

Triggering a Manual Sync

powershell

# Delta sync — syncs only changed objects (fast, use this for most scenarios)
Start-ADSyncSyncCycle -PolicyType Delta

# Full sync — only if schema changed or delta isn't picking up a change
Start-ADSyncSyncCycle -PolicyType Initial

If DDE Sync Stops

Step 1 — Check the ADSync service:

powershell

Get-Service "ADSync"
# If stopped:
Start-Service "ADSync"

Wait 2 minutes, then trigger a delta sync and check status.

Step 2 — Check event logs for errors:

powershell

Get-WinEvent -LogName Application -MaxEvents 50 |
  Where-Object {
    $_.LevelDisplayName -in "Error","Warning" -and
    $_.ProviderName -like "*ADSync*"
  } |
  Select-Object TimeCreated, Message

Look for: authentication errors, disk space issues, connectivity failures.

Step 3 — Test connectivity to DDE Entra:

powershell

Test-NetConnection -ComputerName login.microsoftonline.com -Port 443

If this fails from ENTRACAZC01, the Palo Alto firewall may be blocking outbound HTTPS from the firewall VNet. The firewall must allow ENTRACAZC01 → internet on port 443 for Entra Connect to function.

Step 4 — Credential check:

Entra Connect authenticates to Entra ID using a dedicated service account. If the account password expired or was disabled:

powershell

# Rerun Entra Connect wizard to update credentials
"%ProgramFiles%\Microsoft Azure AD Sync\Bin\miiserver.exe"

Navigate to the configuration wizard → Change user sign-in → re-enter credentials.

Step 5 — Scope of impact while sync is down:

Existing DDE sessions continue — Entra ID caches synced attributes
New logins using password hash sync continue for approximately 72 hours after the last successful sync
Changes made in on-prem AD (new users, group changes, disables) will not propagate to DDE Entra until sync resumes
Customer access may fail if their AD account was modified and sync hasn't replicated the change

DDE-specific consideration: If a Medicare customer account is disabled in on-prem AD (offboarding, suspected breach), that disable will not hit DDE Entra until sync resumes. If immediate revocation is required, disable the account directly in DDE Entra while sync is down, then verify after sync resumes that the on-prem state didn't re-enable it.

Viewing Sync Errors

powershell

# Objects failing to sync
Get-ADSyncCSObject -ConnectorName "<dde-domain>" -ErrorsOnly

Portal: DDE Entra ID → Entra Connect → Sync errors

Common errors in the DDE context:

Error	Cause	Fix
`AttributeValueMustBeUnique`	Duplicate UPN in DDE AD or Entra	Find the duplicate and rename one
`InvalidSoftMatch`	Entra has a cloud-only object conflicting with on-prem	Hard-match using ImmutableId
`PasswordPolicyViolation`	On-prem password doesn't meet DDE Entra policy	Self-resolves on next customer password change

6. DDE Firewall Management

How DDE Firewall Differs from PROD

Attribute	PROD (Azure)	DDE (Azure)
Panorama	No Panorama — managed via local GUI and Terraform	DDE Panorama present (PAN-OS 12.1.2, rebuilt March 2026)
Panorama connection	N/A	Firewall not yet reconnected — pending a scheduled reboot
Access to Panorama	N/A	Twingate → DDE private IP (Keeper: "Panorama — DDE")
Inbound customer traffic	Not customer-facing at this tier	Customer AVD connections enter on untrust zone
User-ID	Not in use for internal servers	Used to attribute session host traffic to individual customers
DNS for FQDNs	Azure-provided DNS	Must use DDE's internal DNS — no Cisco Umbrella

Accessing DDE Panorama

Connect Twingate (if not already connected) — Panorama is on a private IP
Retrieve the Panorama IP and credentials from Keeper → "Panorama — DDE"
Browse to https://<panorama-ip> and accept the self-signed cert warning
Verify connected firewalls under Panorama tab → Managed Devices → Summary

Current known issue: The DDE firewall is not connected to Panorama because a firewall reboot is required to restore the management connection following the March 2026 rebuild. Until reconnected, manage the DDE firewall directly via its local web GUI (same Keeper entry for the firewall management IP and credentials).

When the firewall shows Disconnected in Panorama after reconnection is confirmed:

Verify the management IP is reachable from Panorama
Check the Panorama IP is still configured in the firewall's device settings
Look at the firewall's system logs for the connection attempt: Monitor → Logs → System → filter: ( subtype eq panorama )

DDE Firewall — Local GUI Access

Until Panorama reconnection is complete, use the local GUI:

Connect Twingate
Retrieve the firewall management IP and credentials from Keeper → "Palo Alto — DDE"
Browse to https://<firewall-mgmt-ip>

Policy changes made locally must be reconciled into Panorama once it's reconnected, to avoid configuration drift. Document any local changes made while Panorama is disconnected.

DDE Zone Architecture (Customer Traffic Context)

Internet (customer AVD clients)
    │
    ▼ untrust zone
Palo Alto DDE Firewall
    │
    ▼ trust zone
AVD Session Hosts (Billings VNet)
    │
    ▼ (via VNet peering to Firewall VNet)
Active Directory (ACTDIRAZC01, ACTDIRAZC02)

Key difference: In DDE, inbound traffic from the untrust zone is customer traffic, not attacker traffic. Rules allowing untrust→trust for AVD ports are expected and intentional. The Palo Alto IPS/threat profiles on this traffic still inspect for malicious payloads even in allowed flows.

AVD ports to allow (untrust → trust for customer connections):

TCP 443 (HTTPS/WSS — AVD gateway traffic)
UDP 3478 / UDP 3479 (TURN/STUN for media optimization, optional but improves performance)
TCP 3389 is not opened to the internet — AVD handles the transport

Firewall Rule Changes

Changes to DDE firewall rules follow the same approval process as PROD:

File a change request per security/firewall-change-procedure.md
Rory is the only authorized committer to Panorama or the local firewall GUI
Test on report-only or with specific source IPs before broad rollout
Log all changes — the config log in Panorama (or local Monitor → Logs → Config) is the audit trail

7. DDE Monitoring

How job-orchestrator-cirius Works

job-orchestrator-cirius is an Azure Container Apps Job in rg-logging-logs (logging subscription, PROD tenant — the logging infra lives in PROD, not in DDE). It runs every 4 hours, connects to the DDE tenant via its own managed identity with DDE-scoped role assignments, runs DDE-specific agents, and posts findings to the SecOps platform.

The job lives in the PROD logging subscription, but it monitors the DDE tenant. This is by design — all SecOps jobs share the same logging infrastructure.

Checking job-orchestrator-cirius Execution

Portal (PROD tenant, logging subscription):rg-logging-logs → job-orchestrator-cirius → Execution history

Should show Succeeded executions every 4 hours. A gap longer than 5 hours means the job missed a run.

CLI:

bash

az containerapp job execution list \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --output table \
  --query "sort_by([].{Name:name,StartTime:properties.startTime,Status:properties.status}, &StartTime) | [-10:]"

Reading job-orchestrator-cirius Logs

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-orchestrator-cirius' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

For a failed execution:

bash

# Get the most recent failed execution name
az containerapp job execution list \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "[?properties.status=='Failed'] | [-1].name" \
  --output tsv

Manually Triggering job-orchestrator-cirius

After a fix or after a missed run window:

bash

az containerapp job start \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs

DDE-Specific Alerts and KQL Queries

All DDE logs flow to cirius-logging-law-central (workspace 5d76d1f2, PROD logging subscription) via cross-tenant diagnostic settings. Run these queries from the central LAW.

DDE AVD Connection Events (past 24h):

kql

WVDConnections
| where TimeGenerated > ago(24h)
| where TenantId == "ff1c5d68"
| summarize Sessions = count() by UserName, SessionHostName, State
| sort by Sessions desc

DDE AVD Connection Failures:

kql

WVDErrors
| where TimeGenerated > ago(24h)
| where TenantId == "ff1c5d68"
| project TimeGenerated, UserName, SessionHostName, ServiceError, ErrorMessage
| sort by TimeGenerated desc

DDE Entra Sign-In Failures (customer accounts):

kql

SignInLogs
| where TimeGenerated > ago(24h)
| where TenantId == "ff1c5d68"
| where ResultType != 0
| summarize Failures = count() by UserPrincipalName, ResultDescription, IPAddress
| sort by Failures desc

DDE Break-Glass Activity (any sign-in = immediate alert):

kql

SignInLogs
| where TimeGenerated > ago(30d)
| where UserPrincipalName == "cirius-breakglass@ciriusdde.com"
| project TimeGenerated, UserPrincipalName, ResultType, IPAddress, Location, AppDisplayName
| sort by TimeGenerated desc

DDE Session Host Health Check Failures:

kql

WVDAgentHealthStatus
| where TimeGenerated > ago(1h)
| where TenantId == "ff1c5d68"
| where HealthCheckResult != "HealthCheckSucceeded"
| project TimeGenerated, SessionHostName, HealthCheckName, HealthCheckResult, AdditionalFailureDetails
| sort by TimeGenerated desc

DDE Entra Connect Sync Failures (from on-prem AD events forwarded via Azure Monitor Agent):

kql

Event
| where TimeGenerated > ago(24h)
| where Computer == "ENTRACAZC01"
| where Source contains "ADSync"
| where EventLevelName in ("Error", "Warning")
| project TimeGenerated, EventLevelName, RenderedDescription
| sort by TimeGenerated desc

DDE Azure Activity — Privileged Operations:

kql

AzureActivity
| where TimeGenerated > ago(24h)
| where SubscriptionId in ("<billings-sub-id>", "<firewall-sub-id>", "<logging-sub-id>", "<main-sub-id>")
| where ActivityStatusValue == "Success"
| where OperationNameValue has_any ("delete", "write", "action")
| project TimeGenerated, Caller, OperationNameValue, ResourceGroup, ResourceId
| sort by TimeGenerated desc

DDE-Specific Alert Conditions

These conditions in DDE are higher severity than the equivalent in PROD, given the HIPAA scope:

Condition	Why It Matters in DDE
Session host health check failure	Customers cannot connect; Medicare billing may stall
Entra Connect sync stopped > 1 hour	Customer account changes (disables, password resets) won't propagate
DDE CA policy modified	Could open unauthorized PHI access or lock out customers
Break-glass sign-in	In HIPAA scope — must be documented and justified
More than 5 concurrent user sessions from a single source IP	Possible credential sharing or account compromise
AVD session to untrust zone traffic	Session host may be compromised; outbound PHI exfiltration risk

8. Common DDE Issues

Customer connects, authenticates successfully, but gets a black screen instead of the desktop or app.

Step 1 — Check session host health:

powershell

Get-AzWvdSessionHost -ResourceGroupName <rg-name> -HostPoolName <pool-name> |
  Where-Object { $_.Status -ne "Available" }

If hosts are Unhealthy or NeedsAssistance, the black screen is likely a host issue. Drain the affected host and route the user to another host.

Step 2 — Check FSLogix profile:

Black screens are frequently caused by FSLogix profile mount failures (user's profile storage is unreachable or the profile is locked by a previous session).

On the session host (via RDP to the host itself):

powershell

# Check FSLogix event log
Get-WinEvent -LogName "Microsoft-FSLogix-Apps/Operational" -MaxEvents 20 |
  Select-Object TimeCreated, LevelDisplayName, Message

Look for: storage account unreachable, profile VHD locked, permission denied.

Step 3 — Kill the stuck profile:

If the profile VHD from a previous session is locked:

Identify which session host holds the stale session for this user
Log off the stale session (see Section 3)
Verify the VHD lock is released — check the FSLogix log again
Have the user try connecting again

Step 4 — Check DDE Entra sign-in logs:

If the user's token is valid but the session is black, the app or shell may have failed to launch. Check:

kql

WVDErrors
| where TimeGenerated > ago(1h)
| where TenantId == "ff1c5d68"
| where UserName contains "<username>"
| project TimeGenerated, ErrorMessage, SessionHostName

Application Not Launching in AVD

User can connect to the session but published applications don't open.

Step 1 — Check Application Group assignments:Azure Virtual Desktop → Application Groups → [group] → Applications

Verify the application is still published and the user's account is in the assigned user/group for the Application Group.

Step 2 — Check the application path on the session host:

The published app points to an executable path on the session host. If the software was uninstalled or the path changed:

RDP to the session host
Verify the executable at the path listed in the Application Group exists

Step 3 — MSIX App Attach (if used):

If apps are delivered via MSIX App Attach rather than installed directly:

powershell

# Check MSIX package health on the host
Get-WvdMsixPackage -HostPoolName <pool-name> -ResourceGroupName <rg-name>

The AppAttachHealthCheck in the session host health check results will show failures here.

Step 4 — Profile not loading (app settings missing):

If the app launches but user settings are missing or the app prompts for first-run setup on every login, FSLogix may not be loading the profile. Check the FSLogix log (same as above) and verify the profile share is reachable from the session host.

Session Host Unhealthy / NeedsAssistance

A session host shows Unhealthy, NeedsAssistance, or individual health checks failing.

Step 1 — Drain the host immediately:

powershell

Update-AzWvdSessionHost `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -Name <host-name> `
  -AllowNewSession:$false

This prevents new customer sessions from routing to the degraded host while you investigate.

Step 2 — Review health check details:

Portal: Host Pool → Session Hosts → [host] → Health Checks tab

Each check type reports the specific failure. Common DDE causes:

Health Check Failure	Common Cause in DDE	Action
`DomainJoinedCheck`	Host dropped domain membership	Rejoin to DDE domain or rebuild
`DomainTrustCheck`	Secure channel to ACTDIRAZC01/02 broken	`netdom resetpwd` on host or restart `Netlogon`
`FSLogixHealthCheck`	Profile storage unreachable	Check storage account firewall rules, private endpoint
`MonitoringAgentCheck`	Azure Monitor Agent stopped	Restart the agent service on the host

Step 3 — Check VM-level events:

Via Portal: [VM] → Boot diagnostics → Serial log for catastrophic failures.

Or on the host directly:

powershell

Get-WinEvent -LogName System -MaxEvents 30 |
  Where-Object { $_.LevelDisplayName -in "Error","Critical" } |
  Select-Object TimeCreated, Id, Message

Step 4 — If the host cannot recover:

Ensure all user sessions are logged off

Delete the session host from the host pool:

powershell

Remove-AzWvdSessionHost `
  -ResourceGroupName <rg-name> `
  -HostPoolName <pool-name> `
  -Name <host-name>

Delete or redeploy the underlying VM
Terraform a replacement host (increment count, open a PR) or manually add via portal

Entra Connect Sync Stops — Customer Can't Log In

If the DDE Entra Connect sync fails and a customer's account was recently modified (password change, enable/disable), they may experience authentication failures.

For active customers locked out: reset the password directly in DDE Entra (not on-prem) to unblock immediately
Fix the sync issue following Section 5 procedures
After sync resumes, verify the on-prem and Entra states are consistent

PHI access must not remain blocked pending an infrastructure repair — if a Medicare customer cannot access their billing session, escalate the sync fix as P1.

Customer Reports MFA Prompt Loop

Customer is prompted for MFA repeatedly or cannot complete the MFA challenge.

Check DDE CA policies for sign-in frequency settings — if set too aggressively (e.g., every 1 hour), customers get re-prompted frequently
Check if the customer's authenticator app or phone number is registered in DDE Entra: DDE Entra → Users → [user] → Authentication methods
If no auth method is registered, the customer cannot complete MFA — register a TOTP app or send an auth setup link
Temporary exception (break-glass path only): if a customer is locked out and cannot register, an admin can temporarily exclude their account from the MFA CA policy, complete the auth method setup, then re-include them

9. DDE vs PROD — What's Shared vs Analogous

Nothing is shared. The environments are fully isolated. There are no shared resources, no cross-tenant links, no shared credentials, no shared networks.

What is analogous — same patterns, entirely separate implementations:

Component	PROD Implementation	DDE Implementation
Entra ID	`ciriusgroup.com` tenant	`ciriusdde.com` tenant
Entra Connect server	PROD DC or dedicated sync server	`ENTRACAZC01` in DDE Firewall VNet
Palo Alto firewall	VM-Series in Azure PROD network	VM-Series in DDE Firewall VNet
Panorama	No Panorama (Azure PROD)	DDE Panorama (rebuilt March 2026)
Recovery Services Vault	PROD vault(s)	DDE Firewall Vault + DDE Billings Vault
Log Analytics workspace	`cirius-logging-law-central` (workspace `5d76d1f2`)	Same workspace receives DDE logs via cross-tenant diagnostic settings
SecOps monitoring	`job-orchestrator-cirius`	`job-orchestrator-cirius`
Break-glass account	`cirius-breakglass@ciriusgroup.com`	`cirius-breakglass@ciriusdde.com`
Conditional Access	PROD CA policies	DDE CA policies (separate, HIPAA-aligned)
Key Vault	PROD Key Vault(s)	DDE Key Vault(s) — separate credentials
Terraform state	PROD main subscription	DDE main subscription
GitHub OIDC	PROD federated credential	DDE federated credential
Container Registry	`ciriusagentsprod.azurecr.io`	Same registry (agent images are shared, credentials are separate)

The Log Analytics workspace is the only infrastructure element that receives data from both tenants. The workspace lives in PROD, and DDE diagnostic settings are configured to ship to it cross-tenant. This is intentional — all security telemetry is centralized in one place for the SecOps platform. The workspace does not write back to DDE; it is receive-only from DDE's perspective.

DDE Infrastructure Overview
DDE Network Topology
DDE ASR Configuration
DDE Subscription Security Controls
Entra Connect Operations — PROD Entra Connect reference (same procedures apply to DDE)
Conditional Access Reference — CA policy patterns (PROD-focused, same concepts apply in DDE)
Container Apps Jobs Monitoring — job-orchestrator-cirius follows the same patterns
Panorama Day-to-Day — DDE Panorama operations follow the same GUI layout
Azure Daily Operations — tenant switching, CLI auth
Break-Glass Procedure

Document History

Date	Change	Author
May 2026	Initial draft — full DDE operations guide covering AVD session host management, user sessions, DDE Entra, DDE CA, Entra Connect, firewall, monitoring, common issues, and PROD/DDE comparison.	Rory

DDE Environment Operations ​

1. What DDE Is — Quick Orientation for Someone Who Knows PROD ​

2. AVD Session Host Management ​

Portal Navigation ​

Checking Host Pool Health ​

Draining a Session Host ​

Restarting a Session Host ​

Adding Capacity (Scaling the Host Pool) ​

3. User Session Management ​

Finding Active Sessions ​

Logging Off Stuck Sessions ​

Shadow / Connect to a Session (Admin View) ​

4. DDE-Specific Admin Tasks ​

Managing DDE Entra Users vs PROD Entra Users ​

DDE Conditional Access ​

5. DDE Entra Connect: Sync Health ​

Checking Sync Health ​

Triggering a Manual Sync ​

If DDE Sync Stops ​

Viewing Sync Errors ​

6. DDE Firewall Management ​

How DDE Firewall Differs from PROD ​

Accessing DDE Panorama ​

DDE Firewall — Local GUI Access ​

DDE Zone Architecture (Customer Traffic Context) ​

Firewall Rule Changes ​

7. DDE Monitoring ​

How job-orchestrator-cirius Works ​

Checking job-orchestrator-cirius Execution ​

Reading job-orchestrator-cirius Logs ​

Manually Triggering job-orchestrator-cirius ​

DDE-Specific Alerts and KQL Queries ​

DDE-Specific Alert Conditions ​

8. Common DDE Issues ​

AVD Black Screen After Login ​

Application Not Launching in AVD ​

Session Host Unhealthy / NeedsAssistance ​

Entra Connect Sync Stops — Customer Can't Log In ​

Customer Reports MFA Prompt Loop ​

9. DDE vs PROD — What's Shared vs Analogous ​

Related Documents ​

Document History ​

DDE Environment Operations

1. What DDE Is — Quick Orientation for Someone Who Knows PROD

2. AVD Session Host Management

Portal Navigation

Checking Host Pool Health

Draining a Session Host

Restarting a Session Host

Adding Capacity (Scaling the Host Pool)

3. User Session Management

Finding Active Sessions

Logging Off Stuck Sessions

Shadow / Connect to a Session (Admin View)

4. DDE-Specific Admin Tasks

Managing DDE Entra Users vs PROD Entra Users

DDE Conditional Access

5. DDE Entra Connect: Sync Health

Checking Sync Health

Triggering a Manual Sync

If DDE Sync Stops

Viewing Sync Errors

6. DDE Firewall Management

How DDE Firewall Differs from PROD

Accessing DDE Panorama

DDE Firewall — Local GUI Access

DDE Zone Architecture (Customer Traffic Context)

Firewall Rule Changes

7. DDE Monitoring

How job-orchestrator-cirius Works

Checking job-orchestrator-cirius Execution

Reading job-orchestrator-cirius Logs

Manually Triggering job-orchestrator-cirius

DDE-Specific Alerts and KQL Queries

DDE-Specific Alert Conditions

8. Common DDE Issues

AVD Black Screen After Login

Application Not Launching in AVD

Session Host Unhealthy / NeedsAssistance

Entra Connect Sync Stops — Customer Can't Log In

Customer Reports MFA Prompt Loop

9. DDE vs PROD — What's Shared vs Analogous

Related Documents

Document History