Docs

Appearance

Azure OpenAI Operations Runbook ​

Scope: cirius-openai-prod.openai.azure.com — Azure PROD tenant d477c9f8 (ciriusgroup.com) Key Vault: cirius-openai-kv-prod (logging subscription, rg-logging-logs) LAW: cirius-logging-law-central (ID: 5d76d1f2, rg-logging-logs) HIPAA note: Security event data (not PHI) sent to OpenAI for analysis is covered under the Microsoft BAA.

1. What Is Deployed ​

Deployment ​

FieldValue
Deployment namegpt-4o
Endpointhttps://cirius-openai-prod.openai.azure.com/
Resource grouprg-logging-logs (logging subscription)
ModelGPT-4o
Auth (primary)Managed identity (Cognitive Services OpenAI User role)
Auth (fallback)API key stored in cirius-openai-kv-prod as openai-api-key

Consumers ​

ConsumerIdentityAuth method
17-agent security monitoring system (Container Apps Job)System-assigned managed identity on jobManaged identity (preferred)
kobe-bot (dev testing)User-assigned or system identity on Cloud PC EC2 instanceAPI key fallback via Key Vault

The 17-agent system runs on 4-hour cycles. Agents span categories: entra, network, aws, cortex, supply_chain, operational, inventory. SecurityAgent, AnalystAgent, and DetectionAgent all make LLM calls.

Verify the deployment exists ​

bash
az login --tenant d477c9f8-xxxx-xxxx-xxxx-xxxxxxxxxxxx
az account set --subscription "<logging-subscription-id>"

az cognitiveservices account deployment list \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --output table

Expected output: a row with gpt-4o in the NAME column, Succeeded in PROVISIONING STATE.

2. API Key Management ​

Where keys live ​

API keys are stored in cirius-openai-kv-prod as secret openai-api-key. Agents retrieve the key via their managed identity — the key value never touches environment variables baked into container images or Terraform state.

Retrieve current key value (for debugging) ​

bash
az keyvault secret show \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --query value \
  --output tsv

Retrieve current key metadata (no value exposure) ​

bash
az keyvault secret show \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --query "{id:id, enabled:attributes.enabled, expires:attributes.expires, created:attributes.created}" \
  --output json

Standard key rotation (planned, zero-downtime) ​

Azure OpenAI resources have two keys (KEY1, KEY2). Rotate by cycling one key at a time while the other stays live.

Step 1: Regenerate KEY2 in Azure (KEY1 stays active — no downtime)

bash
az cognitiveservices account keys regenerate \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --key-name key2

# Capture the new KEY2 value
NEW_KEY=$(az cognitiveservices account keys list \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --query key2 \
  --output tsv)

Step 2: Store the new key in Key Vault (creates a new version; old version preserved)

bash
az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "$NEW_KEY"

Step 3: Force Container Apps Job to pick up the new key value

Container Apps using secretsRef do not auto-reload on Key Vault updates. Force a new revision:

bash
# Identify the job resource — adjust name if orchestrator has been split
az containerapp job list \
  --resource-group rg-logging-logs \
  --output table

# Force job restart (next scheduled execution picks up new secret)
az containerapp job start \
  --name <job-name> \
  --resource-group rg-logging-logs

Step 4: Verify agents are authenticating successfully

Run a manual job execution and check logs:

bash
az containerapp job logs show \
  --name <job-name> \
  --resource-group rg-logging-logs \
  --tail 100

Watch for 401 Unauthorized from cirius-openai-prod.openai.azure.com — that indicates the new key was not picked up correctly.

Step 5: Update Key Vault to point to KEY1 (if needed) and disable old KEY2

If agents previously used KEY1, you now switch them to KEY2 (just stored), then regenerate KEY1:

bash
az cognitiveservices account keys regenerate \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --key-name key1

# Store new KEY1 if it's the canonical secret
NEW_KEY1=$(az cognitiveservices account keys list \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --query key1 \
  --output tsv)

az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "$NEW_KEY1"

Step 6: Disable old Key Vault versions

bash
# List all versions
az keyvault secret list-versions \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --output table

# Disable each old version by its GUID
az keyvault secret set-attributes \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --version <old-version-guid> \
  --enabled false

Emergency rotation (key suspected or confirmed compromised) ​

Do this in order. Speed matters. Do not wait for a change window.

bash
# STEP 1: Regenerate BOTH keys immediately at the source
# This invalidates the compromised key within seconds.
az cognitiveservices account keys regenerate \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --key-name key1

az cognitiveservices account keys regenerate \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --key-name key2

# STEP 2: Capture the new key1
NEW_KEY=$(az cognitiveservices account keys list \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --query key1 \
  --output tsv)

# STEP 3: Store new key in Key Vault
az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "$NEW_KEY"

# STEP 4: Force job restart
az containerapp job start \
  --name <job-name> \
  --resource-group rg-logging-logs

# STEP 5: Disable ALL previous Key Vault versions
az keyvault secret list-versions \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --output tsv --query "[].{id:id, enabled:attributes.enabled}"

# For each old version:
az keyvault secret set-attributes \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --version <version-guid> \
  --enabled false

After the immediate rotation (within 24 hours):

  • File an internal incident in SecOps (POST /api/incidents)
  • Document the exposure window start date
  • Query Key Vault audit logs for all accesses to openai-api-key during the exposure window (KQL in section 7)
  • Query Azure OpenAI diagnostic logs for API calls during the window — look for unusual callers, unusual endpoints, or volume spikes
  • Confirm whether the key was in any git commit, log line, or application config; if so, assume full exposure and document

3. Quota Monitoring ​

What quotas apply ​

Azure OpenAI enforces two quotas per deployment:

Quota typeMeaningWhat happens when hit
TPM (Tokens Per Minute)Total token throughput429 TooManyRequests — agents back off
RPM (Requests Per Minute)Request rate429 TooManyRequests — agents back off

Quotas are per-deployment (not per key). The gpt-4o deployment shares quota across all callers.

Check current quota allocation in the portal ​

Portal path: Azure Portal → Cognitive Services → cirius-openai-prod → Model deployments → gpt-4o → Edit deployment

The slider shows the allocated TPM cap. Default allocation at time of deployment may be lower than the subscription limit — this is intentional.

Check quota allocation via CLI ​

bash
az cognitiveservices account deployment show \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --deployment-name gpt-4o \
  --query "sku" \
  --output json

The capacity field is in units of 1,000 TPM. A value of 10 means 10,000 TPM allocated.

Check subscription-level quota limits ​

bash
# List all quota for GPT-4o models in the region
az cognitiveservices usage list \
  --location eastus \
  --output table

This shows the total quota available to the subscription vs what is committed across all deployments.

Check real-time TPM/RPM usage in the portal ​

Portal path: Azure Portal → Cognitive Services → cirius-openai-prod → Metrics

Select metric: Token Based Usage — shows prompt tokens and completion tokens. Split by ModelDeploymentName to isolate gpt-4o.

Select metric: Requests — shows request count. Filter by StatusCode = 429 to see quota-exceeded calls.

What happens when quota is hit ​

  • All agents return 429 TooManyRequests
  • Azure OpenAI includes a Retry-After header in the 429 response (seconds to wait)
  • Agents should respect this header — check agents/ code to confirm retry logic is implemented
  • If agents do not back off, repeated 429s waste context and drive up error rates without making progress
  • The 4-hour cycle will still complete if enough of the quota is recovered within the window

Request a quota increase ​

Portal path: Azure Portal → Cognitive Services → cirius-openai-prod → Model deployments → gpt-4o → Edit deployment → drag TPM slider → Save

If the slider is maxed (subscription limit reached):

  1. Go to Azure Portal → Help + Support → New support request
  2. Issue type: Service and subscription limits (quotas)
  3. Quota type: Cognitive Services
  4. Request: increase TPM for GPT-4o in the relevant region
  5. Justify with: current allocation, actual peak usage from Metrics, business purpose

Quota increase requests for Azure OpenAI typically take 1-5 business days. Request 2x current peak to leave headroom.

4. Token Usage and Cost Monitoring ​

Cost model ​

Azure OpenAI bills per token, split between input (prompt) and output (completion). GPT-4o pricing at time of writing:

  • Input: ~$5.00 per 1M tokens
  • Output: ~$15.00 per 1M tokens

With 17 agents running every 4 hours (6 cycles/day), calculate expected daily spend against actual billing.

View usage in the portal ​

Portal path: Azure Portal → Cognitive Services → cirius-openai-prod → Metrics

Key metrics to watch:

MetricUse
Token Based UsageInput + output tokens per time window
RequestsCall volume; split by status code to see 429 ratio
Total RequestsAggregate call count

Set the time range to 24h and granularity to 1h. Normal pattern is 6 spikes per day aligned with the 4-hour job cycle. Flat lines between cycles = correct. Continuous usage = an agent is stuck in a loop or an unauthorized caller is present.

View costs in Azure Cost Management ​

Portal path: Azure Portal → Cost Management + Billing → Cost analysis → filter by resource cirius-openai-prod

Group by: Day, Resource, Meter. Compare current month vs last month.

KQL — Token usage from LAW diagnostics ​

Prerequisite: diagnostic settings on cirius-openai-prod forwarding to cirius-logging-law-central. Verify:

bash
az monitor diagnostic-settings list \
  --resource "$(az cognitiveservices account show --name cirius-openai-prod --resource-group rg-logging-logs --query id --output tsv)" \
  --output table

If not configured:

bash
LAW_ID=$(az monitor log-analytics workspace show \
  --workspace-name cirius-logging-law-central \
  --resource-group rg-logging-logs \
  --query id \
  --output tsv)

OPENAI_ID=$(az cognitiveservices account show \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --query id \
  --output tsv)

az monitor diagnostic-settings create \
  --name openai-to-law \
  --resource "$OPENAI_ID" \
  --workspace "$LAW_ID" \
  --logs '[{"category": "RequestResponse", "enabled": true}, {"category": "Audit", "enabled": true}]' \
  --metrics '[{"category": "AllMetrics", "enabled": true}]'

KQL — hourly token usage by deployment

kql
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where OperationName == "ChatCompletions_Create"
| where TimeGenerated > ago(24h)
| extend PromptTokens = toint(parse_json(requestBody_s).usage.prompt_tokens)
| extend CompletionTokens = toint(parse_json(requestBody_s).usage.completion_tokens)
| summarize
    TotalPromptTokens = sum(PromptTokens),
    TotalCompletionTokens = sum(CompletionTokens),
    TotalTokens = sum(PromptTokens + CompletionTokens),
    RequestCount = count()
    by bin(TimeGenerated, 1h)
| sort by TimeGenerated asc

KQL — 429 error rate over last 24 hours

kql
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(24h)
| summarize
    TotalRequests = count(),
    QuotaErrors = countif(httpStatusCode_d == 429),
    AuthErrors = countif(httpStatusCode_d == 401 or httpStatusCode_d == 403),
    Timeouts = countif(httpStatusCode_d == 408 or httpStatusCode_d == 503)
    by bin(TimeGenerated, 1h)
| extend ErrorRate = round(100.0 * QuotaErrors / TotalRequests, 1)
| sort by TimeGenerated asc

KQL — usage per agent cycle (normal vs abnormal)

Normal pattern: 17 agents each call the API during a 4-hour window. Expect 6 burst windows per day. Each agent makes 1-10 calls per cycle depending on findings volume.

kql
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where OperationName == "ChatCompletions_Create"
| where TimeGenerated > ago(7d)
| summarize RequestCount = count(), TotalTokens = sum(toint(parse_json(requestBody_s).usage.total_tokens))
    by bin(TimeGenerated, 4h)
| sort by TimeGenerated asc

Abnormal patterns to investigate:

PatternLikely cause
Continuous requests outside cycle windowsAgent stuck in retry loop, unauthorized caller, or job schedule misconfigured
Token volume 3x+ normal for a cycleA prompt is growing unbounded (no truncation), or an agent is processing a data spike
Zero requests for 8+ hoursJob failed to start, authentication broke, network issue
429 rate > 10% sustainedQuota too low; multiple agents competing at cycle peak

5. Deployment Management ​

List current deployments ​

bash
az cognitiveservices account deployment list \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --output table

Show deployment details (model version, capacity) ​

bash
az cognitiveservices account deployment show \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --deployment-name gpt-4o \
  --output json

Add a new deployment ​

Use case: adding a second model (e.g., gpt-4o-mini for lower-cost agent tasks).

bash
az cognitiveservices account deployment create \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --deployment-name gpt-4o-mini \
  --model-name gpt-4o-mini \
  --model-version "2024-07-18" \
  --model-format OpenAI \
  --sku-capacity 10 \
  --sku-name Standard

After creating, update the relevant agent code to reference the new deployment name and test before deploying to production.

Update a model version ​

Azure OpenAI supports in-place model version upgrades. Versions are named (e.g., 2024-05-13, 2024-08-06). Upgrading changes the model behavior — test first.

bash
az cognitiveservices account deployment create \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --deployment-name gpt-4o \
  --model-name gpt-4o \
  --model-version "2024-08-06" \
  --model-format OpenAI \
  --sku-capacity 10 \
  --sku-name Standard

Note: this is effectively a redeploy of the same deployment name with a new version. Azure will update in place. Agents calling the deployment name (gpt-4o) will automatically use the new version on next call.

Check model deprecation timeline ​

Portal path: Azure Portal → Cognitive Services → cirius-openai-prod → Model deployments → gpt-4o — if the version column shows a warning icon, a deprecation date is set.

bash
# List available model versions to see what's current
az cognitiveservices account list-models \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --output table

Microsoft publishes deprecation timelines at: https://learn.microsoft.com/azure/ai-services/openai/concepts/model-retirements

Check this quarterly. Deprecated models are removed by Microsoft with typically 60-90 days notice. Update deployment version before the deadline or agents will start returning 404 DeploymentNotFound.

6. Access Control ​

Auth model ​

Preferred: managed identity with Cognitive Services OpenAI User RBAC role on the Azure OpenAI resource. No key ever touches the code.

Fallback: API key from Key Vault, fetched at runtime via the same managed identity with Key Vault Secrets User on cirius-openai-kv-prod.

RBAC roles on the Azure OpenAI resource ​

RoleCan doWho has it
Cognitive Services OpenAI UserCall the inference API (completions, chat, embeddings)Container Apps Job managed identities, kobe-bot
Cognitive Services OpenAI ContributorCall API + manage deployments, fine-tunesRory's account for management tasks
Cognitive Services ContributorFull CRUD on the resource including keysBreak-glass only
ReaderView resource metadata, no API callsAudit tools, Terraform plan runs

List current role assignments on the Azure OpenAI resource ​

bash
OPENAI_ID=$(az cognitiveservices account show \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs \
  --query id \
  --output tsv)

az role assignment list \
  --scope "$OPENAI_ID" \
  --include-inherited \
  --output table

Grant managed identity access to call the API ​

bash
# Get the Container Apps Job managed identity principal ID
PRINCIPAL_ID=$(az containerapp job show \
  --name <job-name> \
  --resource-group rg-logging-logs \
  --query "identity.principalId" \
  --output tsv)

az role assignment create \
  --assignee "$PRINCIPAL_ID" \
  --role "Cognitive Services OpenAI User" \
  --scope "$OPENAI_ID"

Grant kobe-bot API key read access (dev) ​

kobe-bot accesses the API key via Key Vault. The bot's identity needs Key Vault Secrets User on cirius-openai-kv-prod:

bash
KV_ID=$(az keyvault show \
  --name cirius-openai-kv-prod \
  --query id \
  --output tsv)

# Replace with kobe-bot's object ID (user or service principal)
az role assignment create \
  --assignee "<kobe-bot-object-id>" \
  --role "Key Vault Secrets User" \
  --scope "$KV_ID"

Audit who has called the API (via diagnostics in LAW) ​

kql
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where OperationName == "ChatCompletions_Create"
| where TimeGenerated > ago(30d)
| summarize CallCount = count()
    by CallerIPAddress, identity_claim_upn_s, identity_claim_oid_g
| sort by CallCount desc

For managed identities, identity_claim_upn_s is empty. Use identity_claim_oid_g (object ID) and cross-reference with:

bash
az ad sp show --id <object-id> --query displayName --output tsv

7. Troubleshooting ​

429 — Quota exceeded ​

Symptoms: Agents log 429 TooManyRequests. Job takes longer than normal. Findings are delayed.

Check:

  1. Portal path: Cognitive Services → cirius-openai-prod → Metrics → filter by StatusCode=429, last 1h. If the 429 rate is continuous and not just at cycle start, quota is too low for load.

  2. KQL in LAW:

kql
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where httpStatusCode_d == 429
| where TimeGenerated > ago(1h)
| summarize Count = count() by bin(TimeGenerated, 5m)
| sort by TimeGenerated asc

Resolution:

  • Short-term: stagger agent execution so they do not all call the API simultaneously at cycle start
  • Medium-term: request quota increase (section 3)
  • If 429s are from a single agent type running in a loop: check agent code for retry without backoff

401 — Unauthorized (authentication failure) ​

Symptoms: Agents log 401 Unauthorized. All agents fail, not just one.

Triage path:

  1. Is the agent using managed identity or API key?

    • Managed identity: check that the role assignment still exists (section 6 — az role assignment list). Check that the Container Apps Job still has an identity assigned.
    • API key: check that openai-api-key is enabled in Key Vault and the current version is not disabled. Check that the job's managed identity can still read the secret.

  2. Test managed identity auth from a shell inside the job container:

bash
# From inside the container (az CLI must be available)
az login --identity
az cognitiveservices account keys list \
  --name cirius-openai-prod \
  --resource-group rg-logging-logs

If this returns a 403, the managed identity lacks the role. Reassign it.

  1. Test Key Vault secret retrieval:
bash
az keyvault secret show \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --query value \
  --output tsv

If this fails with Forbidden, the identity lacks Key Vault Secrets User. If it succeeds but agents still get 401, the key value stored in Key Vault is stale — the key was regenerated at the source and not updated in Key Vault.

403 — Forbidden (authorized identity, missing permission) ​

Symptoms: The identity authenticates successfully but cannot call the inference endpoint.

Check: The identity has a valid Azure AD token but lacks Cognitive Services OpenAI User on the deployment. This is a missing RBAC assignment, not an auth failure.

bash
az role assignment list \
  --scope "$OPENAI_ID" \
  --include-inherited \
  --assignee "<principal-id>" \
  --output table

If the role is missing, add it (section 6). Role assignments propagate within 1-2 minutes.

Timeout / slow responses ​

Symptoms: Agents take longer than expected. Occasional 408 or 504 responses. Job cycle overruns the 4-hour window.

Check:

  1. Model load: Azure OpenAI shows increased latency under high regional load. Check Azure status page: https://status.azure.com for Cognitive Services incidents in the deployed region.

  2. Token count: Large prompts take longer to process. If an agent is sending very large context windows (security log dumps without truncation), latency increases. Check the token counts in LAW diagnostics — if prompt tokens per request jumped significantly, an agent changed behavior.

  3. KQL — P95 latency by hour:

kql
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where OperationName == "ChatCompletions_Create"
| where TimeGenerated > ago(24h)
| extend DurationMs = toint(DurationMs)
| summarize
    P50 = percentile(DurationMs, 50),
    P95 = percentile(DurationMs, 95),
    P99 = percentile(DurationMs, 99)
    by bin(TimeGenerated, 1h)
| sort by TimeGenerated asc

Normal P95 for GPT-4o with typical security analysis prompts: 3-15 seconds. If P95 exceeds 30 seconds consistently, the model is under regional load or prompts are too large.

Resolution:

  • Regional load: no action needed unless it's affecting the job completing within its cycle
  • Prompt size: cap input tokens in agent code (set max_tokens on requests, truncate log context before sending)
  • If the cycle is not completing: reduce the scheduled cycle from 4 hours or increase parallelism

KQL — Errors in LAW (all types, last 24h) ​

kql
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where httpStatusCode_d >= 400
| where TimeGenerated > ago(24h)
| summarize
    Count = count()
    by httpStatusCode_d, OperationName, ResultType, ResultDescription
| sort by Count desc

8. Monitoring Alerts ​

Alert thresholds ​

ConditionThresholdSeverityAction
429 errors sustained> 20 per 5 min for 15 minHighInvestigate quota; consider staggering agents
401/403 errorsAny in productionCriticalAuth broke; agents not running; immediate triage
Zero requests during expected cycle window0 requests for 5+ hoursHighJob failed to start or agents crashing before API call
Token usage spike> 3x rolling 7-day average for a cycleMediumAgent prompt growing unbounded or data volume spike
Latency P95> 60 seconds sustained over 30 minMediumRegional degradation or prompt size issue

Create an alert for sustained 429s ​

Portal path: Azure Monitor → Alerts → Create alert rule → Scope: cirius-openai-prod

Or via CLI:

bash
# Create a metric alert for 429 quota errors
az monitor metrics alert create \
  --name "openai-quota-exceeded" \
  --resource-group rg-logging-logs \
  --scopes "$OPENAI_ID" \
  --condition "count 'Requests' > 20 where StatusCode = 429" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 2 \
  --description "Azure OpenAI quota exceeded — 20+ 429 errors in 5 minutes" \
  --action <action-group-id>

Create an alert for zero requests (agents not calling API) ​

This requires a KQL-based scheduled alert in LAW since a metric going to zero is harder to alert on via metric alerts:

Portal path: LAW workspace cirius-logging-law-central → Alerts → New alert rule → Custom log search

kql
// Alert fires if fewer than 5 API calls in the last 5 hours
// Run this check every 1 hour
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(5h)
| summarize RequestCount = count()
| where RequestCount < 5

Configure: evaluation every 1 hour, alert if result count > 0 (i.e., query returns a row), severity 2.

Create an alert for auth failures ​

bash
# Metric alert on any 401/403 responses
az monitor metrics alert create \
  --name "openai-auth-failures" \
  --resource-group rg-logging-logs \
  --scopes "$OPENAI_ID" \
  --condition "count 'Requests' > 0 where StatusCode = 401" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 1 \
  --description "Azure OpenAI auth failures — agents cannot authenticate" \
  --action <action-group-id>

Action group ​

Alerts should route to the existing action group used for infrastructure alerts. If one does not exist:

bash
az monitor action-group create \
  --name "ag-infra-alerts" \
  --resource-group rg-logging-logs \
  --short-name infra \
  --email-receivers name=rory address=rgarshol@gmail.com

Reference the action group ID in alert creation commands above.

Quick Reference ​

TaskCommand
List deploymentsaz cognitiveservices account deployment list --name cirius-openai-prod --resource-group rg-logging-logs --output table
Show deployment configaz cognitiveservices account deployment show --name cirius-openai-prod --resource-group rg-logging-logs --deployment-name gpt-4o
List API keysaz cognitiveservices account keys list --name cirius-openai-prod --resource-group rg-logging-logs --output table
Regenerate key1az cognitiveservices account keys regenerate --name cirius-openai-prod --resource-group rg-logging-logs --key-name key1
Show KV secret metadataaz keyvault secret show --vault-name cirius-openai-kv-prod --name openai-api-key --query "{id:id,enabled:attributes.enabled,expires:attributes.expires}"
List role assignmentsaz role assignment list --scope <openai-resource-id> --include-inherited --output table
Check diagnostic settingsaz monitor diagnostic-settings list --resource <openai-resource-id>
Force job executionaz containerapp job start --name <job-name> --resource-group rg-logging-logs
Tail job logsaz containerapp job logs show --name <job-name> --resource-group rg-logging-logs --tail 100

Internal use only — Cirius Group