Network Connectivity Troubleshooting

Who this is for: Any engineer who received a "can't reach X" ticket and needs a structured path from symptom to resolution. Start at the symptom that matches, follow the decision tree, run the diagnostic commands, escalate when indicated.

Environment summary:

Azure PROD tenant d477c9f8 — ciriusgroup.com, primary production
Azure DDE tenant ff1c5d68 — ciriusdde.com, AVD/published apps to Medicare
AWS 7-account org — DR and services
Remote access: Twingate (all users), GlobalProtect (CEO/CTO only)
Firewalls: 4x Palo Alto VM-Series, Panorama management, DNS Security active
Hub-spoke VNets: all spoke-to-spoke traffic forced through Palo Alto NVA via UDR
LAW workspace: cirius-logging-law-central (ID 5d76d1f2, RG rg-logging-logs)
MDR: Arctic Wolf

Quick Triage — Which Layer Is Broken?

Before diving into a specific section, answer these three questions in order:

Can the user reach the public internet?
- No → start at Section 1 — Twingate
- Yes → continue
Is this an internal hostname that resolves to nothing?
- Yes → start at Section 3 — Azure Internal DNS or Section 4 — Cloudflare DNS
- No → continue
Does the hostname resolve but TCP/UDP never connects?
- Yes → start at Section 2 — Palo Alto Drops, Section 5 — VNet/UDR, or Section 6 — NSG

1. Twingate Connectivity

1.1 User Cannot Connect to Twingate at All

Symptom: Client shows "Not connected", "Network unavailable", or Twingate icon is grey/red.

IF the user just enrolled a new device → check Intune compliance first. Twingate blocks non-compliant devices at the trust-level gate.

Check: Intune portal → Devices → find the device → Compliance status
IF "Not compliant" → fix the Intune compliance issue first. Twingate access will not work until the device is compliant.
IF "Compliant" → continue below.

IF the device is compliant but the client still won't connect:

Confirm the network name is cirius (not cirius.twingate.com, not blank)
Ask the user to sign out of the Twingate client and sign back in using their @ciriusgroup.com Microsoft account — token refresh sometimes fixes a stale auth state
Check whether the user's Entra account is enabled: Azure Portal → Entra ID → Users → find user → Account enabled = Yes
Check whether the user's Twingate-mapped Entra group is populated:
- Open Twingate admin portal: https://auth.twingate.com
- Users → find the user → confirm they appear and show the correct group
- If missing: add them to the correct Entra group (AllStaff, PrivilegedUsers, or ContractorAccess) and trigger a manual sync: Settings → Identity Provider → Sync Now

1.2 Connector Down

Symptom: Twingate admin portal shows one or both connectors as offline. Users may report intermittent drops or complete failure depending on whether both connectors are down.

Decision tree:

Are both connectors (vm-twingate-conn01-* AND vm-twingate-conn02-*) down?
├── YES → treat as P1, notify Rory immediately. Full remote access is severed.
└── NO (one down) → one connector handles full load. Non-urgent, but investigate.

Steps to investigate a down connector:

In Azure portal, locate the connector VM (search vm-twingate-conn in the PROD subscription):
- Check VM power state: should be "Running". If "Stopped (deallocated)", start it.
- Check the last heartbeat time in Twingate admin portal → Remote Networks → Connectors

If the VM is running but the connector is still offline, SSH to the VM via Twingate (you need the other connector to be up) or via Azure Bastion:

bash

# Check the Twingate connector service
sudo systemctl status twingate-connector

# If stopped, start it
sudo systemctl start twingate-connector

# View recent logs
sudo journalctl -u twingate-connector --since "1 hour ago" -n 100

Common causes from the logs:
- authentication failed → the connector token is expired or revoked. Regenerate the token in Twingate admin portal → Remote Networks → Connectors → the affected connector → Generate Token, then update it on the VM:
  bash
```
sudo twingate setup --headless --token=<new-token>
sudo systemctl restart twingate-connector
```
- network unreachable → the VM cannot reach *.twingate.com on 443/TCP. Check NSG on the connector subnet (see Section 6) and check the PublicFW traffic logs (see Section 2).
- certificate error → clock skew. Check: timedatectl — NTP should be in sync.
After restoring a connector, verify in Twingate admin portal that status returns to "Online" within 2–3 minutes.

1.3 User Can Connect to Twingate but Cannot Reach a Specific Resource

Symptom: Twingate is connected, internet works, but app.internal.ciriusgroup.com or a specific IP is unreachable.

Step 1 — Is the resource defined in Twingate?

Twingate admin portal → Resources → search for the hostname or IP. If it is not listed, the resource has not been published. Either the resource was not yet added or was recently decommissioned.

IF resource exists in Twingate → continue to Step 2
IF resource does NOT exist → add it per twingate-operations.md, or escalate if you do not know what it should map to

Step 2 — Does the user's group have access to this resource?

Twingate admin portal → Resources → click the resource → Access tab. Confirm the user's group (AllStaff, PrivilegedUsers, or ContractorAccess) is listed. If not, add the appropriate group after confirming with Rory.

Step 3 — Can you reach the resource from a connector?

SSH into one of the Twingate connector VMs via Azure Bastion:

bash

# Test TCP connectivity from the connector to the resource
nc -zv <resource-ip> <port> -w 5

# Test DNS from the connector
nslookup <hostname> <dns-server-ip>

IF nc succeeds from the connector but the user cannot reach it → Twingate config problem (wrong IP, wrong port in resource definition)
IF nc fails from the connector → the path from the connector to the resource is broken → continue to Sections 2, 5, or 6

Step 4 — KQL: Check Twingate connector logs forwarded to LAW

kql

// Twingate resource access failures — last 1 hour
Syslog
| where TimeGenerated > ago(1h)
| where Computer has "twingate"
| where SyslogMessage has "denied" or SyslogMessage has "failed" or SyslogMessage has "error"
| project TimeGenerated, Computer, SyslogMessage
| sort by TimeGenerated desc

2. Palo Alto Firewall Drops

Symptom: Traffic does not flow. nc, curl, or application connections time out or immediately reset. Twingate is up and the path exists in theory, but packets are not passing.

All Palo Alto log analysis should start in Panorama → Monitor → Logs → Traffic or via the CLI on the specific firewall. Determine which firewall is in the path first.

Source in DMZ (10.5.0.0/16) to trust-prod (10.20.0.0/16)?
└── Path: DMZ → PrivateFW → trust-prod. Check PrivateFW.

Source is untrust (internet) to DMZ or trust?
└── Path: internet → PublicFW → (PrivateFW). Check PublicFW first.

Source is trust-spoke to trust-spoke (e.g., identity to prod)?
└── Path: spoke → UDR → PrivateFW → UDR → spoke. Check PrivateFW.

Source is AWS VPC to Azure VNet?
└── Path: AWS VPC → AWS Palo Alto → IPSec → Azure PublicFW → internal. Check both firewalls.

2.1 Traffic Not Flowing — Basic Drop Analysis

On Panorama or via Panorama SSH to the firewall:

bash

# Show active traffic for a specific source and destination (run on the firewall, not Panorama)
show session all filter source <src-ip> destination <dst-ip>

# Show interface statistics
show interface all

# Check if a specific 5-tuple would hit a policy drop (test without sending real traffic)
test security-policy-match source <src-ip> destination <dst-ip> destination-port <port> protocol <6|17>

Panorama GUI — Traffic log filter for drops:

# In Panorama Monitor → Logs → Traffic, use this filter:
(addr.src in <src-ip>) and (addr.dst in <dst-ip>) and (action eq deny)

# Broader: all denies in the last 15 minutes across any source/dest
(action eq deny) and (receive_time geq '2024/01/01 00:00:00')
# (adjust the time to "last 15min" using the Panorama time picker instead)

KQL: Pull Palo Alto traffic logs from LAW:

kql

// Palo Alto denied traffic — last 1 hour
CommonSecurityLog
| where TimeGenerated > ago(1h)
| where DeviceVendor == "Palo Alto Networks"
| where Activity == "TRAFFIC"
| where AdditionalExtensions has "action=deny"
| project TimeGenerated, SourceIP, DestinationIP, DestinationPort, Protocol, 
    DeviceAction, Reason=AdditionalExtensions
| sort by TimeGenerated desc

2.2 Zone Policy — Traffic Denied by Zone-Based Security Policy

Symptom: Log shows action=deny, rule=intrazone-default or rule=interzone-default, or a named deny rule. The path crosses a zone boundary that has no allow rule.

Step 1 — Identify the zones involved:

Zone reference (from network-topology.md):
- untrust         → internet / anything not Cirius-owned
- dmz             → 10.5.0.0/16 (Twingate connectors, DMZ services)
- dmz-auth        → 10.11.0.0/16 (RODC)
- trust-identity  → 10.10.0.0/16 (writable DCs, Entra Connect)
- trust-prod      → 10.20.0.0/16 (prod servers, PHI)
- trust-dev       → 10.30.0.0/16 (dev workloads)
- trust-legacy    → 10.4.0.0/16 (decom June 2026)

Step 2 — Check if a matching allow rule exists in Panorama:

Panorama → Policies → Security → filter by source zone / destination zone. Look for a rule that should cover this traffic. If no rule exists, this requires a firewall change request to Rory — do not create the rule yourself.

Step 3 — If you believe a rule should already cover this traffic:

bash

# On the firewall:
test security-policy-match source <src-ip> destination <dst-ip> destination-port <port> protocol 6

# If it returns "deny" on a rule you expected to allow, check:
# 1. Is the source/destination in the correct address object?
# 2. Is the service object covering the right port?
# 3. Is the rule disabled?

DO NOT create or modify firewall rules. File a request with Rory per security/firewall-change-procedure.md.

2.3 URL Category Block

Symptom: HTTP/HTTPS traffic to a specific URL is blocked. User sees a Palo Alto block page ("This site has been blocked") or connection drops on port 443 only for a specific FQDN.

Step 1 — Check Panorama URL Filtering logs:

Panorama → Monitor → Logs → URL Filtering:

# Filter for the blocked URL
(url contains "example-blocked-site.com") and (action eq block)

Step 2 — Identify the category:

The log line will show the URL category (e.g., malware, phishing, unknown, parked). The URL Filtering profile in use will block certain categories by default.

Step 3 — Check DNS Security:

DNS Security can block resolution before a connection is even attempted. Check:

Panorama → Monitor → Logs → Threat → filter by:

(subtype eq dns)

Look for DNS Security events matching the destination domain.

KQL — Palo Alto threat/URL block events:

kql

CommonSecurityLog
| where TimeGenerated > ago(1h)
| where DeviceVendor == "Palo Alto Networks"
| where Activity in ("THREAT", "URL")
| where DestinationHostName has "domain-to-check"
| project TimeGenerated, SourceIP, DestinationHostName, DestinationPort, 
    Activity, DeviceAction, AdditionalExtensions
| sort by TimeGenerated desc

Step 4 — Resolution path:

IF the category is clearly wrong (legitimate business site marked as malware):
└── Submit a URL recategorization request to Palo Alto: https://urlfiltering.paloaltonetworks.com
    AND raise the URL as a potential allow-list exception to Rory.

IF the category is accurate (site IS malware/phishing):
└── Do not bypass. Escalate to Rory if a business justification exists. Notify Arctic Wolf if
    the user was trying to reach it from a phishing email or unexpected context.

IF DNS Security is blocking a domain that should be allowed:
└── Raise to Rory. DNS Security exceptions require Panorama policy change — do not add yourself.

3. Azure Internal DNS Failures

Symptom: An internal hostname (e.g., vm-prod-app01.ciriusgroup.internal, or a private endpoint like myservice.privatelink.blob.core.windows.net) resolves to nothing or to the wrong IP.

3.1 Identify the DNS Zone Type

Is the hostname ending in *.ciriusgroup.internal or *.ciriusdde.internal?
└── This is a custom Azure Private DNS zone. See Section 3.2.

Is the hostname ending in *.privatelink.*.core.windows.net or *.privatelink.*.azure.com?
└── This is a Private Endpoint DNS zone. See Section 3.3.

Is the hostname something else entirely that should resolve internally?
└── Check if it is a Windows DNS (on-prem AD) record — contact Kevin or Greg.

3.2 Azure Private DNS Zone

Step 1 — Confirm the zone exists and the record is present:

bash

az network private-dns zone list --output table
az network private-dns record-set list -g <resource-group> -z ciriusgroup.internal --output table
az network private-dns record-set a show -g <resource-group> -z ciriusgroup.internal -n <hostname>

Step 2 — Confirm the zone is linked to the VNet:

Private DNS zones only resolve for VNets that are explicitly linked. A missing link is the most common cause of private DNS failures.

bash

az network private-dns link vnet list -g <resource-group> -z ciriusgroup.internal --output table

IF the VNet of the failing VM is not in the list:
└── The VNet link is missing. Add it:
    az network private-dns link vnet create \
      -g <resource-group> \
      -z ciriusgroup.internal \
      -n <link-name> \
      -v <vnet-resource-id> \
      -e false
    (Use -e true to enable auto-registration if this VNet's VMs should auto-register)

Step 3 — Confirm the DNS resolver is reachable from the VM:

Azure VMs use the Azure-provided DNS resolver (168.63.129.16) by default. If a custom DNS server is configured on the VNet, that server must be operational and forwarding queries correctly.

bash

# On the failing VM (or via Azure Bastion / serial console):
nslookup <hostname> 168.63.129.16

# If that works but the VM's normal resolver doesn't:
cat /etc/resolv.conf           # Linux — check what resolver is configured
ipconfig /all | findstr DNS    # Windows — check DNS server list

KQL — Azure DNS resolver failures:

kql

// Azure Diagnostics for DNS resolver errors (if DNS resolver diagnostic logging is enabled)
AzureDiagnostics
| where TimeGenerated > ago(1h)
| where ResourceType == "DNSPRIVATERESOLVERVIRTUALNETWORKLINKS" or 
        Category == "DNSQueryLogs"
| where ResultCode != "NOERROR"
| project TimeGenerated, QueryName = column_ifexists("queryName_s", ""), 
    ResultCode = column_ifexists("resultCode_s", ""), 
    ClientSubnet = column_ifexists("clientSubnet_s", "")
| sort by TimeGenerated desc

3.3 Private Endpoint DNS Resolution

Symptom: A storage account, Key Vault, or other Azure PaaS resource is not resolving to its private endpoint IP — it resolves to the public IP instead, meaning traffic bypasses the private endpoint and may be blocked by the resource's network rules.

Step 1 — Confirm the private endpoint exists:

bash

az network private-endpoint list --output table
# Find the endpoint for your service and note its NIC IP
az network private-endpoint show -n <endpoint-name> -g <resource-group> \
  --query "customDnsConfigs" --output table

Step 2 — Confirm the privatelink DNS zone has the A record:

bash

az network private-dns zone list --output table | grep privatelink
az network private-dns record-set a list -g <resource-group> \
  -z privatelink.blob.core.windows.net --output table

Step 3 — Confirm the privatelink zone is linked to the VNet the failing VM is in:

Same as Section 3.2 Step 2, but for the privatelink.* zone. This is the most frequent cause of private endpoint resolution failures — the zone exists and has the record, but the VNet link is missing.

Step 4 — Verify the resolved IP is the private endpoint IP:

bash

# From the VM:
nslookup myaccount.blob.core.windows.net

# Expected: should return the private endpoint IP (e.g., 10.20.x.x), not 52.x.x.x
# IF it returns the public IP → DNS zone link is missing or the wrong resolver is in use

4. Cloudflare DNS Issues

Symptom: A public hostname (something under ciriusgroup.com or ciriusdde.com) does not resolve, resolves to the wrong IP, or resolves correctly but connections are refused.

4.1 DNS Resolution Not Working Publicly

Step 1 — Confirm the record exists in Cloudflare:

bash

# Check the live DNS response from Cloudflare's resolver
dig @1.1.1.1 <hostname>.ciriusgroup.com A
dig @8.8.8.8 <hostname>.ciriusgroup.com A

# Check what Cloudflare has configured (requires cloudflare CLI or API):
# Via Cloudflare dashboard: dash.cloudflare.com → ciriusgroup.com zone → DNS → Records

Step 2 — Compare with Terraform state:

All Cloudflare DNS is managed via Terraform in azure-infra. If a record is missing, it was either never added or was accidentally deleted. Check the Terraform state and recent PRs.

Step 3 — Check TTL / propagation:

If a record was just changed, it may not have propagated. Default TTL is 300 seconds (5 min) for most records. Check dig +ttl output. If TTL is near zero, wait for the next cycle.

4.2 DNS Security Blocking a Public Domain

Symptom: A public site that should be reachable is blocked by Cloudflare DNS Security (Gateway). User sees a Cloudflare block page or gets NXDOMAIN for a legitimate domain.

Step 1 — Confirm Gateway is in the path:

This only applies to devices enrolled in Cloudflare Gateway (WARP client or a Cloudflare-filtered DNS resolver). Check whether the user is on a WARP-enrolled device.

Step 2 — Check Cloudflare Gateway logs:

Cloudflare dashboard → Zero Trust → Gateway → Activity log. Filter by the user or the domain. The log will show whether Gateway blocked the request and the reason (policy match, category, or security risk classification).

Step 3 — Resolution:

IF the domain is a legitimate business site miscategorized as malicious:
└── Zero Trust → Gateway → Policies → Add a DNS allow policy for the specific domain
    OR submit a recategorization request to Cloudflare.

IF the domain is legitimately blocked (malware, C2, known bad):
└── Do not bypass. Notify Arctic Wolf if the user was directed there via phishing or a
    suspicious process.

4.3 Proxied Record (Orange Cloud) Causing Issues

Symptom: A service behind Cloudflare proxy returns unexpected errors, wrong TLS cert, or connections time out. The symptom varies: could be 524 (timeout), 521 (origin refused), or TLS handshake failures.

Step 1 — Check if the record is proxied:

In Cloudflare dashboard → DNS → Records, look for the orange cloud icon. If proxied, traffic routes through Cloudflare's edge before reaching the origin.

Step 2 — Identify the error code:

Cloudflare Error	Meaning	First Check
521	Origin refused connection	Origin server is down or not listening on the port Cloudflare is hitting
522	Connection timed out to origin	Origin IP unreachable — check NSG, firewall rules on the origin
523	Origin unreachable	Origin IP is unreachable from Cloudflare's network
524	Timeout after connection established	Origin took too long to respond — app issue, not network
526	Invalid SSL cert on origin	Origin's cert is expired, self-signed, or mismatched (when Full Strict is set)

Step 3 — Temporarily test with DNS-only:

If you need to bypass the proxy to test the origin directly, switch the record to DNS-only (grey cloud) in Cloudflare. Coordinate with Rory before doing this on any production record — removing the proxy removes WAF and DDoS protection.

5. VNet Peering / UDR Routing Problems

Symptom: A VM in one spoke VNet cannot reach a VM in another spoke VNet, or spoke-to-spoke traffic is not passing through the Palo Alto NVA as expected. traceroute shows traffic going to a next hop that is not the firewall, or traffic simply stops at the VNet boundary.

Architecture Reminder

In the Cirius hub-spoke model, spoke VNets do not peer directly with each other. All spoke-to-spoke traffic must traverse the Palo Alto NVA (PrivateFW) via User Defined Routes. A routing problem at this layer means either the UDR is missing, the next-hop points to the wrong IP, or VNet peering is broken.

Expected path (trust-prod → trust-identity):
  vm-prod (10.20.x.x)
    → UDR on prod subnet: 0.0.0.0/0 → PrivateFW NIC IP
    → PrivateFW (10.98.x.x)
    → UDR on identity subnet: 0.0.0.0/0 → PrivateFW NIC IP
    → vm-dc (10.10.x.x)

5.1 Check the Effective Routes on the VM NIC

This is the single fastest diagnostic for routing problems:

bash

az network nic show-effective-route-table \
  -n <nic-name> -g <resource-group> \
  --output table

IF the route for the destination subnet shows "VirtualAppliance" nextHopType and the correct 
   PrivateFW NIC IP as nextHopIpAddress → routing is configured correctly. Problem is elsewhere.

IF the route shows "VNetPeering" or "VNetLocal" and routes directly to the destination 
   WITHOUT going through the NVA → UDR is missing or not associated with this subnet.

IF the route shows "Invalid" or is missing entirely → UDR or peering is broken.

5.2 Check the UDR Association

bash

# List route tables in the subscription
az network route-table list --output table

# Show routes in a specific route table
az network route-table show -n <route-table-name> -g <resource-group> \
  --query "routes" --output table

# Confirm the route table is associated with the correct subnet
az network vnet subnet show \
  -g <resource-group> \
  --vnet-name <vnet-name> \
  -n <subnet-name> \
  --query "routeTable" --output table

IF route table is not associated with the subnet:
└── Associate it:
    az network vnet subnet update \
      -g <resource-group> \
      --vnet-name <vnet-name> \
      -n <subnet-name> \
      --route-table <route-table-name>

IF route table exists but the route for the destination is missing or has wrong next hop:
└── This is a Terraform drift or a manual change that was reverted. 
    Do not fix manually — raise to Rory and check git history on azure-infra for recent route table changes.

5.3 Check VNet Peering State

UDR-based routing requires the VNet peering to be in a connected state. A peering that is "Disconnected" breaks routing even if the UDR is correct.

bash

az network vnet peering list \
  --vnet-name <vnet-name> \
  -g <resource-group> \
  --output table

IF peeringState shows "Connected" AND useRemoteGateways / allowForwardedTraffic are correct:
└── Peering is healthy. Problem is elsewhere.

IF peeringState shows "Disconnected" or "Initiated":
└── The peering is broken or was never completed on the remote side.
    Check the peering from the remote VNet side as well:
    az network vnet peering list --vnet-name <remote-vnet> -g <remote-rg> --output table
    IF the remote side also shows "Initiated" → both sides need to be re-confirmed. Raise to Rory.

5.4 Verify Traffic Is Actually Reaching the NVA

Use Azure Network Watcher Next Hop to confirm the platform-level routing decision:

bash

az network watcher show-next-hop \
  --resource-group <resource-group> \
  --vm <source-vm-name> \
  --source-ip <source-ip> \
  --dest-ip <destination-ip>

The output shows the nextHopType and nextHopIpAddress Azure will actually use. If it does not show the PrivateFW NIC as the next hop, the UDR is not in effect.

KQL — Check if Azure route table changes were made recently:

kql

AzureActivity
| where TimeGenerated > ago(24h)
| where OperationNameValue has_any ("Microsoft.Network/routeTables/write", 
    "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write",
    "Microsoft.Network/routeTables/routes/write")
| project TimeGenerated, Caller, OperationNameValue, ResourceGroup, 
    Resource, ActivityStatusValue
| sort by TimeGenerated desc

6. Azure NSG Blocks

Symptom: A specific port or protocol is refused between two resources in Azure. Traffic stays within the VNet (no firewall in the path), but the destination port never responds. Test-NetConnection returns TcpTestSucceeded: False on the expected port.

6.1 Check Effective NSG Rules on the Destination NIC

This is the fastest way to see what rules are actually applying to a VM:

bash

az network nic list-effective-nsg \
  -n <destination-nic-name> \
  -g <resource-group> \
  --output table

Look for a deny rule matching the source IP range and the destination port. NSG rules are evaluated in priority order (lower number = higher priority). A deny at priority 200 beats an allow at priority 300.

Azure Network Watcher — IP Flow Verify:

This tells you whether a specific 5-tuple would be allowed or denied by the NSG on a VM:

bash

az network watcher test-ip-flow \
  --resource-group <resource-group> \
  --vm <vm-name> \
  --direction Inbound \
  --protocol TCP \
  --local <vm-ip>:<port> \
  --remote <source-ip>:<source-port>

Output: "Access: Allow" → NSG is not the problem. Look at Palo Alto (Section 2) or app config.
Output: "Access: Deny" and shows the NSG rule name → that rule is blocking traffic.

6.2 Identify and Fix the Blocking NSG Rule

bash

# Show all rules on a specific NSG
az network nsg show -n <nsg-name> -g <resource-group> \
  --query "securityRules" --output table

# Show the default (built-in) rules too
az network nsg show -n <nsg-name> -g <resource-group> \
  --query "defaultSecurityRules" --output table

IF the block is from a built-in deny rule (priority 65500 "DenyAllInBound"):
└── No higher-priority allow rule exists for this traffic. Add an allow rule or 
    raise to Rory if a firewall policy change is the right fix instead.

IF the block is from a custom rule added by someone:
└── Check the rule name, description, and when it was added (see KQL below).
    Confirm with Rory before removing it — it may be intentional.

KQL — Recent NSG rule changes:

kql

AzureActivity
| where TimeGenerated > ago(7d)
| where OperationNameValue has "Microsoft.Network/networkSecurityGroups"
| where ActivityStatusValue == "Success"
| project TimeGenerated, Caller, OperationNameValue, ResourceGroup, Resource
| sort by TimeGenerated desc

6.3 NSG Flow Logs — Check Whether Traffic Was Logged as Denied

If NSG flow logs are enabled (they should be, routed to LAW):

kql

// NSG flow log — denied flows involving a specific IP
AzureNetworkAnalytics_CL
| where TimeGenerated > ago(1h)
| where FlowStatus_s == "D"   // D = Denied
| where SrcIP_s == "<source-ip>" or DestIP_s == "<destination-ip>"
| project TimeGenerated, SrcIP_s, DestIP_s, DestPort_d, Protocol_s, 
    FlowStatus_s, NSGRule_s = NSGList_s
| sort by TimeGenerated desc

7. Diagnostic Commands Reference

Azure CLI — Quick Reference

bash

# Check VM NIC effective routes
az network nic show-effective-route-table -n <nic> -g <rg> --output table

# Check effective NSG on a NIC
az network nic list-effective-nsg -n <nic> -g <rg> --output table

# Test IP flow through NSG
az network watcher test-ip-flow \
  --resource-group <rg> --vm <vm> \
  --direction Inbound --protocol TCP \
  --local <ip>:<port> --remote <ip>:<port>

# Network Watcher next-hop
az network watcher show-next-hop --resource-group <rg> \
  --vm <vm> --source-ip <src> --dest-ip <dst>

# Check VNet peering state
az network vnet peering list --vnet-name <vnet> -g <rg> --output table

# Check Private DNS zone links
az network private-dns link vnet list -g <rg> -z <zone> --output table

# Run LAW query via CLI
az monitor log-analytics query --workspace 5d76d1f2 \
  --analytics-query "<kql>" --output table

Palo Alto CLI — Quick Reference

Run these on the firewall via SSH or Panorama CLI passthrough (request cli target <device-name>):

bash

# Show traffic session for a 5-tuple
show session all filter source <ip> destination <ip> destination-port <port>

# Test policy match without sending traffic
test security-policy-match source <src> destination <dst> \
  destination-port <port> protocol 6

# Show interface summary (confirm interfaces are up)
show interface all

# Show routing table
show routing route

# Show HA state (confirm both firewalls in an HA pair are healthy)
show high-availability state

# Show system resource usage
show system resources

# Test DNS Security lookup
test dns-security category fqdn <domain>

# Show counters — look for drops
show counter global filter severity drop

# Packet capture on an interface (use sparingly in prod)
debug dataplane packet-diag set filter on
debug dataplane packet-diag set filter match source <ip> destination <ip>
debug dataplane packet-diag clear log log
debug dataplane packet-diag set log feature flow basic
# ... reproduce the issue ...
debug dataplane packet-diag show log log

Twingate Admin Portal Checks

Remote Networks → Connectors — confirm both conn01 and conn02 show Online
Users → find user → Groups — confirm group assignment is correct
Resources → find resource → Access — confirm user's group has access
Resources → find resource → verify IP/FQDN and port are correct
Settings → Identity Provider → Last synced — trigger Sync Now if stale

Azure Network Watcher Tools

In Azure portal: Network Watcher → your region (West US 2 for PROD, check per-account for AWS-connected resources)

Tool	Use For
IP Flow Verify	Is a specific 5-tuple blocked by an NSG?
Next Hop	What will Azure route this packet to? (confirms UDR is working)
Connection Troubleshoot	End-to-end connectivity test between two Azure VMs
VPN Diagnostics	Diagnose site-to-site VPN tunnel (for AWS ↔ Azure tunnels)
NSG Flow Logs	Historical record of allowed/denied flows per NSG

KQL — All-in-One Network Connectivity Investigation

kql

// Summary: any network-related events for a specific source IP in the last 2 hours
let src_ip = "<source-ip>";
let dst_ip = "<destination-ip>";
union
(
    CommonSecurityLog
    | where TimeGenerated > ago(2h)
    | where DeviceVendor == "Palo Alto Networks"
    | where SourceIP == src_ip or DestinationIP == dst_ip
    | project TimeGenerated, Source="PaloAlto", SourceIP, DestinationIP, 
        DestinationPort, DeviceAction, Activity
),
(
    AzureNetworkAnalytics_CL
    | where TimeGenerated > ago(2h)
    | where SrcIP_s == src_ip or DestIP_s == dst_ip
    | project TimeGenerated, Source="NSGFlowLog", SourceIP=SrcIP_s, 
        DestinationIP=DestIP_s, DestinationPort=DestPort_d, DeviceAction=FlowStatus_s, 
        Activity=NSGList_s
),
(
    AzureActivity
    | where TimeGenerated > ago(2h)
    | where OperationNameValue has "network"
    | project TimeGenerated, Source="AzureActivity", SourceIP=CallerIpAddress, 
        DestinationIP="", DestinationPort=0, DeviceAction=ActivityStatusValue, 
        Activity=OperationNameValue
)
| sort by TimeGenerated desc

8. Escalation Paths

Handle Internally (No Arctic Wolf Involvement Required)

Scenario	Owner	Action
Twingate connector service crashed	Rory	Restart connector service, check token
Azure Private DNS zone missing VNet link	Rory	Add VNet link via CLI
NSG deny rule blocking known-good traffic	Rory	Confirm intent, modify rule if approved
UDR not associated with a subnet (drift)	Rory	Restore association, check Terraform
VNet peering disconnected	Rory	Re-establish peering, check Azure Health
Cloudflare DNS record missing or wrong	Rory	Fix via Terraform PR or emergency dashboard change
Twingate resource missing or wrong IP	Rory	Update resource in Twingate admin portal
Palo Alto interface/HA issue	Rory	Check HA state, contact Palo Alto TAC if needed
Firewall policy missing a needed rule	Rory	File change request, Rory implements in Panorama

Escalate to Arctic Wolf

Contact Arctic Wolf via the Concierge portal (credentials in Keeper → Vendor Accounts → Arctic Wolf). For P1, call the emergency line immediately — do not wait for ticket creation.

Scenario	Priority	Why Arctic Wolf
Palo Alto DNS Security blocking a domain that turns out to be C2	P1	Active malware activity — AW has MDR authority to contain
DNS queries from internal hosts going to unexpected external resolvers	P1/P2	Possible DNS hijack or malware C2 exfil
Twingate connector repeatedly authenticating from unexpected IPs	P2	Possible token theft or connector compromise
NSG or firewall logs show lateral movement patterns (e.g., port scans from an internal IP)	P1/P2	Active threat — AW has act-first authority
A user reports their Twingate session was used from a location they don't recognize	P2	Account compromise, not a config issue
DNS Security repeatedly blocking a domain that a process (not user) is querying	P2	Malware reaching out to C2; escalate even before confirming
Traffic logs show outbound data to unknown IPs from a server that should have no outbound internet	P1	Data exfil pattern

Arctic Wolf has act-first authority. When you open a P1 or P2 ticket, they may isolate devices or block IPs before you respond. That is intentional. Do not open a ticket asking them to "just monitor for now" on a P1 scenario — tell them what you know and let them work.

Escalate to Rory (Even if You Think You Have a Fix)

Any change to Panorama policy, address objects, or security profiles — Rory is the sole approver
Any Twingate resource that serves Tier 0 or Tier 1 access
Any routing change that touches the NVA path (UDR, VNet peering between hub and spoke)
Any DNS change that touches email records (MX, DKIM, SPF, DMARC) — breaking these causes immediate mail delivery failures

Escalate to Azure Support

Azure platform issue (VNet peering broken with no local config change)
Private DNS resolver returning wrong results with no zone config change
Network Watcher shows "Unknown" as next-hop type with no clear cause
Open a P1 (Sev A) ticket if production is down with no identifiable fix

Twingate Operations — full Twingate admin guide
Panorama Day-to-Day — firewall log navigation and Panorama GUI
DNS and Cloudflare Operations — DNS record management
Firewall Rule Reference — governance and rule change process
KQL Query Reference — full library of LAW queries
Arctic Wolf Day-to-Day — MDR relationship and ticket handling
Incident Response — full IR playbook for security events
Network Topology — IP ranges, zones, firewall placement

Network Connectivity Troubleshooting ​

Quick Triage — Which Layer Is Broken? ​

1. Twingate Connectivity ​

1.1 User Cannot Connect to Twingate at All ​

1.2 Connector Down ​

1.3 User Can Connect to Twingate but Cannot Reach a Specific Resource ​

2. Palo Alto Firewall Drops ​

2.1 Traffic Not Flowing — Basic Drop Analysis ​

2.2 Zone Policy — Traffic Denied by Zone-Based Security Policy ​

2.3 URL Category Block ​

3. Azure Internal DNS Failures ​

3.1 Identify the DNS Zone Type ​

3.2 Azure Private DNS Zone ​

3.3 Private Endpoint DNS Resolution ​

4. Cloudflare DNS Issues ​

4.1 DNS Resolution Not Working Publicly ​

4.2 DNS Security Blocking a Public Domain ​

4.3 Proxied Record (Orange Cloud) Causing Issues ​

5. VNet Peering / UDR Routing Problems ​

Architecture Reminder ​

5.1 Check the Effective Routes on the VM NIC ​

5.2 Check the UDR Association ​

5.3 Check VNet Peering State ​

5.4 Verify Traffic Is Actually Reaching the NVA ​

6. Azure NSG Blocks ​

6.1 Check Effective NSG Rules on the Destination NIC ​

6.2 Identify and Fix the Blocking NSG Rule ​

6.3 NSG Flow Logs — Check Whether Traffic Was Logged as Denied ​

7. Diagnostic Commands Reference ​

Azure CLI — Quick Reference ​

Palo Alto CLI — Quick Reference ​

Twingate Admin Portal Checks ​

Azure Network Watcher Tools ​

KQL — All-in-One Network Connectivity Investigation ​

8. Escalation Paths ​

Handle Internally (No Arctic Wolf Involvement Required) ​

Escalate to Arctic Wolf ​

Escalate to Rory (Even if You Think You Have a Fix) ​

Escalate to Azure Support ​

Related Runbooks ​

Network Connectivity Troubleshooting

Quick Triage — Which Layer Is Broken?

1. Twingate Connectivity

1.1 User Cannot Connect to Twingate at All

1.2 Connector Down

1.3 User Can Connect to Twingate but Cannot Reach a Specific Resource

2. Palo Alto Firewall Drops

2.1 Traffic Not Flowing — Basic Drop Analysis

2.2 Zone Policy — Traffic Denied by Zone-Based Security Policy

2.3 URL Category Block

3. Azure Internal DNS Failures

3.1 Identify the DNS Zone Type

3.2 Azure Private DNS Zone

3.3 Private Endpoint DNS Resolution

4. Cloudflare DNS Issues

4.1 DNS Resolution Not Working Publicly

4.2 DNS Security Blocking a Public Domain

4.3 Proxied Record (Orange Cloud) Causing Issues

5. VNet Peering / UDR Routing Problems

Architecture Reminder

5.1 Check the Effective Routes on the VM NIC

5.2 Check the UDR Association

5.3 Check VNet Peering State

5.4 Verify Traffic Is Actually Reaching the NVA

6. Azure NSG Blocks

6.1 Check Effective NSG Rules on the Destination NIC

6.2 Identify and Fix the Blocking NSG Rule

6.3 NSG Flow Logs — Check Whether Traffic Was Logged as Denied

7. Diagnostic Commands Reference

Azure CLI — Quick Reference

Palo Alto CLI — Quick Reference

Twingate Admin Portal Checks

Azure Network Watcher Tools

KQL — All-in-One Network Connectivity Investigation

8. Escalation Paths

Handle Internally (No Arctic Wolf Involvement Required)

Escalate to Arctic Wolf

Escalate to Rory (Even if You Think You Have a Fix)

Escalate to Azure Support

Related Runbooks