SecOps Agent Pipeline — Troubleshooting Runbook

⚠️ Job names updated June 2026. The 3-job architecture this runbook was written for was consolidated; commands below have been re-pointed at job-orchestrator-cirius (plus job-mini-orchestrator-cirius, job-secops-maintenance-cirius). Sections that loop over "all three jobs" are redundant but harmless.

Scope: 17-agent security monitoring pipeline running as Azure Container Apps Jobs
Registry: ciriusagentsprod.azurecr.io
Resource group: rg-logging-logs (logging subscription)
SecOps platform: secops.bedrockcybersecurity.org
LAW workspace: cirius-logging-law-central (ID 5d76d1f2)
Azure OpenAI: gpt-4o at cirius-openai-prod.openai.azure.com

Architecture at a Glance

Three Container Apps Jobs, each running independently on a 4-hour schedule:

Job	Environment	Agent Coverage
`job-orchestrator-cirius`	Azure PROD (`d477c9f8`)	Entra, network, operational, inventory, kill chain
`job-orchestrator-cirius`	Azure DDE (`ff1c5d68`)	DDE Entra, AVD, Panorama, DDE AD
`job-orchestrator-cirius`	AWS 7-account org	CloudTrail, Security Hub, IAM, GuardDuty

Run sequence within each job:

Orchestrator (orchestrator.py) starts; discovers agent modules in its subdirectory
Each mini-agent: queries source → builds findings list → POSTs to /api/incidents
SecurityAgent: fetches all NEW incidents → checks known-good rules → sets CLOSED or OPEN
Analyst (Detection) agent: enriches OPEN tickets with LLM triage via Azure OpenAI
Heartbeat email sent; Telegram notification only when SecurityAgent flags urgent findings

Shared dependencies (failure here affects all three jobs):

Key Vault cirius-openai-kv-prod — SecOps API key and OpenAI key
Container registry ciriusagentsprod.azurecr.io — image pull
SecOps platform secops.bedrockcybersecurity.org — findings destination

1. Finding Job Execution History

Azure Portal

Portal → Resource groups → rg-logging-logs → [job name] → Execution history

Each row shows start time, duration, and status (Succeeded / Failed / Running). A healthy job runs every 4 hours. Any Failed status requires investigation. A gap longer than 5 hours (or 90 minutes once PROD moves to hourly) means a missed run.

To identify which 4-hour cycle an execution corresponds to: Container Apps Job schedules run at 0 */4 * * * UTC, meaning executions fire at 00:00, 04:00, 08:00, 12:00, 16:00, 20:00 UTC. The start time in execution history is UTC — convert to Pacific (UTC-7 PDT / UTC-8 PST) as needed.

CLI — List Recent Executions

bash

# Last 10 executions with status, sorted by start time
az containerapp job execution list \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "sort_by([].{Name:name,Start:properties.startTime,Status:properties.status}, &Start) | [-10:]" \
  --output table

Replace job-orchestrator-cirius with job-orchestrator-cirius or job-orchestrator-cirius as needed.

CLI — Find the Last Failed Execution

bash

az containerapp job execution list \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "[?properties.status=='Failed'] | [-1].{Name:name,Start:properties.startTime,End:properties.endTime}" \
  --output json

Note the Start and End values — use them to bound the log query in Section 2.

CLI — Check All Three Jobs at Once

bash

for JOB in job-orchestrator-cirius job-orchestrator-cirius job-orchestrator-cirius; do
  echo "=== $JOB ===" && az containerapp job execution list --name $JOB --resource-group rg-logging-logs --query "sort_by([].{Start:properties.startTime,Status:properties.status}, &Start) | [-3:]" --output table
done

How to Read Execution Status

Status	Meaning	Action
`Succeeded`	Job ran to completion, all agents exited normally	Verify findings in SecOps UI
`Failed`	Job exited with non-zero code; some or all agents did not complete	Read logs immediately
`Running`	Job is currently executing	Monitor; check again in a few minutes
No recent entries	Job has not been triggered — scheduler may be broken	Check trigger config (Section 4)

2. Reading Agent Logs

Portal — Log Stream (Live)

For a currently running job: Portal → rg-logging-logs → [job name] → [execution name] → Replicas → [replica] → Log stream

Log stream only works while the execution is running. For completed executions, use Log Analytics.

Log Analytics — General Log Query

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-orchestrator-cirius' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

Log Analytics — Logs for a Specific Execution

After getting the start time from the execution list (Section 1), query that window:

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-orchestrator-cirius' | where TimeGenerated between (datetime('2026-05-25T04:00:00Z') .. datetime('2026-05-25T04:45:00Z')) | sort by TimeGenerated asc" \
  --output table

Replace the datetime values with the execution's actual start and end times.

KQL — Agent Errors in the Last 24 Hours

Run this in Portal → cirius-logging-law-central → Logs:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(24h)
| where ContainerAppName_s in ("job-orchestrator-cirius", "job-orchestrator-cirius", "job-orchestrator-cirius")
| where Log_s has_any ("ERROR", "Exception", "Traceback", "CRITICAL", "failed", "error")
| project TimeGenerated, ContainerAppName_s, Log_s
| sort by TimeGenerated desc
| take 100

KQL — Last Activity per Job (Detect Missing Runs)

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(12h)
| where ContainerAppName_s in ("job-orchestrator-cirius", "job-orchestrator-cirius", "job-orchestrator-cirius")
| summarize LastLog = max(TimeGenerated) by ContainerAppName_s
| extend MinutesSinceLastLog = datetime_diff('minute', now(), LastLog)
| sort by MinutesSinceLastLog desc

If MinutesSinceLastLog exceeds 240 (the 4-hour interval), that job has not logged anything recently and either missed its run or the scheduler is not firing.

UTC Time Correlation

Container Apps logs are in UTC. Execution history timestamps are UTC. The job schedule (0 */4 * * *) fires at UTC midnight, 04:00, 08:00, 12:00, 16:00, 20:00. Convert to local time (Pacific: UTC-7 PDT in summer, UTC-8 PST in winter) when correlating with local incident reports.

3. Diagnosing Specific Failure Modes

3.1 LAW Query Timeout or Throttle (429 / HttpResponseError)

Symptoms in logs:

azure.core.exceptions.HttpResponseError: (ThrottledRequest) Too many requests for workspace
azure.core.exceptions.HttpResponseError: Request timeout exceeded

Root cause investigation:

Check if the failure is isolated to LAW-dependent agents (PROD and DDE jobs query LAW; AWS does not):

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s in ("job-orchestrator-cirius", "job-orchestrator-cirius")
| where Log_s has "HttpResponseError" or Log_s has "ThrottledRequest"
| project TimeGenerated, ContainerAppName_s, Log_s
| sort by TimeGenerated desc

Check if PROD and DDE jobs are running simultaneously — concurrent LAW queries from both jobs contend on the same workspace. Stagger schedules to offset by 30 minutes:
- PROD: 0 */4 * * *
- DDE: 30 */4 * * *
Identify the specific query that timed out. The log line before the exception will show the KQL being executed. Run that query manually in Log Analytics (Section 7) to see if it's over-broad.

Fix procedure:

Throttle (429): Wait and manually trigger the job after 10–15 minutes. If recurring, check if the LAW workspace is at capacity, reduce query time ranges in agent code, or stagger job schedules.
Query timeout: The agent's KQL is too expensive — reduce the time window or add filters. Fix in bedrock-soc repo and redeploy.

Verify fix: Check the next execution's logs for a clean run with no HttpResponseError. Confirm findings appear in SecOps after the run.

3.2 Azure OpenAI Rate Limit

Symptoms in logs:

openai.RateLimitError: Rate limit exceeded: gpt-4o
openai.APIStatusError: 429

The Detection agent (Analyst LLM triage) and SecurityAgent both call Azure OpenAI. Rate limit errors from either will appear in logs after the mini-agents have completed their runs.

Root cause investigation:

Check if the failure is at SecurityAgent or Detection agent:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s in ("job-orchestrator-cirius", "job-orchestrator-cirius", "job-orchestrator-cirius")
| where Log_s has "RateLimitError" or Log_s has "429" or Log_s has "openai"
| project TimeGenerated, ContainerAppName_s, Log_s

Check the Azure OpenAI usage in the portal: Portal → cirius-openai-prod → Monitoring → Metrics → Token usage. If the deployment is consistently at its TPM quota, the limit needs to be raised.
If only one of the three jobs is hitting limits, but all three run on the same 4-hour schedule, concurrent OpenAI calls from multiple jobs may be exhausting the quota together.

Fix procedure:

Short-term: Manually trigger the failed job once the quota resets (typically within a minute for per-minute limits, or wait for the next hour for hourly limits).
Long-term: Increase the token-per-minute (TPM) quota on the gpt-4o deployment in the Azure OpenAI portal, or add retry logic with exponential backoff in agent code.

Verify fix: On the next run, look for successful LLM calls:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(2h)
| where ContainerAppName_s == 'job-orchestrator-cirius'
| where Log_s has "SecurityAgent" or Log_s has "analyst"
| project TimeGenerated, Log_s
| sort by TimeGenerated asc

3.3 SecOps API Unreachable

Symptoms in logs:

ConnectionRefused: https://secops.bedrockcybersecurity.org
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected)
httpx.ConnectError: [Errno 111] Connection refused
502 Bad Gateway

Root cause investigation:

Check SecOps platform health immediately (Section 6 of this document).
Check if all three jobs are failing simultaneously — if so, the platform is down (not an agent issue):
bash
```
curl -s -o /dev/null -w "%{http_code}" https://secops.bedrockcybersecurity.org/health
```
If only one job is affected and others can reach SecOps, there may be a transient network issue or the Container App for that job lost egress.

Fix procedure: Agents are designed to fail-open: they log the error and continue to the next agent rather than stopping the entire run. This means a SecOps outage during a job run results in a partial run — agents completed their queries but findings were never posted.

Restore the SecOps platform (see SecOps Platform Deployment).

Manually trigger all three jobs to post the findings that were missed:

bash

az containerapp job start --name job-orchestrator-cirius --resource-group rg-logging-logs
az containerapp job start --name job-orchestrator-cirius --resource-group rg-logging-logs
az containerapp job start --name job-orchestrator-cirius --resource-group rg-logging-logs

Note that a cold-starting SecOps Container App may take 30–60 seconds to become ready. Agents may hit connection errors during the cold-start window even when the platform is healthy. If this is the cause, a single retry on the next scheduled run resolves it.

Verify fix: After triggering, check SecOps for new findings:

bash

curl -s -H "X-API-Key: $SECOPS_API_KEY" "https://soc.bedrockcybersecurity.org/api/incidents?created_since=1h&limit=20" | jq '.[].created_at'

3.4 Key Vault Secret Fetch Failure

Symptoms in logs:

AzureKeyVaultSecretsMissingException
azure.core.exceptions.ClientAuthenticationError
azure.core.exceptions.HttpResponseError: 403 Forbidden
SecretNotFound: The secret 'openai-api-key' was not found

Key Vault failures at job startup block all agents in that job — no secrets means no API key, which means nothing can run.

Root cause investigation:

Identify whether it is an authentication failure (managed identity issue) or a secret not found:
- ClientAuthenticationError / 403 → managed identity role assignment missing
- SecretNotFound → secret name mismatch or secret was deleted

Check managed identity role assignments for the job:

bash

# Get the managed identity client ID for the job
az containerapp job show \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "identity.userAssignedIdentities" \
  --output json

# List role assignments for that identity
az role assignment list \
  --assignee <client-id-from-above> \
  --output table

The identity must have Key Vault Secrets User on cirius-openai-kv-prod.

Verify the secret exists and is enabled:

bash

az keyvault secret show \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --query "{enabled:attributes.enabled, expires:attributes.expires}" \
  --output table

If enabled is false or the secret has expired, the fetch will fail.

Fix procedure:

Missing role assignment: Add Key Vault Secrets User role via Terraform (file a PR) or ad-hoc for immediate remediation:

bash

az role assignment create \
  --assignee <managed-identity-client-id> \
  --role "Key Vault Secrets User" \
  --scope "/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.KeyVault/vaults/cirius-openai-kv-prod"

Secret expired or disabled: Re-enable or rotate the secret:

bash

az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "<new-value>"

Secret name mismatch: Check agent code for the exact secret name expected. It must match exactly what is stored in Key Vault.

Verify fix: Manually trigger the job and watch the startup logs. A successful Key Vault fetch appears before any agent output. No KeyVault error lines within the first 30 seconds of execution means the fetch succeeded.

3.5 AWS Credential Expiration

Symptoms in logs (job-orchestrator-cirius only):

botocore.exceptions.ClientError: An error occurred (ExpiredTokenException)
botocore.exceptions.NoCredentialsError: Unable to locate credentials
botocore.exceptions.TokenRetrievalError: Error retrieving OIDC token

Root cause investigation: AWS credentials are delivered via environment variables or OIDC token federation — not via Azure managed identity. The OIDC token has a short TTL (typically 1 hour). If the job's OIDC federation is misconfigured or the token cannot be refreshed, AWS API calls fail.

Check how AWS credentials are configured on the job:

bash

az containerapp job show \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "properties.template.containers[0].env[?contains(name, 'AWS')].{Name:name,Value:value}" \
  --output table

If using environment variable credentials (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY): check whether the IAM user's key has been rotated or deactivated in AWS IAM.
If using OIDC federation: verify the AWS IAM role trust policy still trusts the correct GitHub org and repository:
- Required subject format: repo:Cirius-Group-Inc/ops-automation:ref:refs/heads/main
- Any deviation (wrong repo name, wrong branch) causes token rejection.

Fix procedure:

Expired static credentials: Rotate the IAM access key in AWS IAM, update the Container Apps Job environment variables (via Terraform PR), and store the new key in Key Vault rather than as a plain env var.
OIDC token failure: Verify the IAM role trust policy and Container Apps federation configuration match. Check that the GitHub org name is Cirius-Group-Inc (not CiriusGroup — OIDC breaks on the mismatch).

Verify fix: Trigger job-orchestrator-cirius manually and look for successful AWS API calls in logs (absence of ClientError / NoCredentialsError in the first agent's output).

3.6 Container Image Pull Failure

Symptoms in logs / Azure Portal:

Failed to pull image "ciriusagentsprod.azurecr.io/secops-agents:latest"
unauthorized: authentication required
manifest unknown: manifest tagged by "v1.2.3" is not found

The job cannot start at all — no agents run, no findings are posted, job shows Failed immediately.

Root cause investigation:

Verify the Container Apps Job has AcrPull on the container registry:

bash

az role assignment list \
  --scope "/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.ContainerRegistry/registries/ciriusagentsprod" \
  --output table

Look for AcrPull assigned to the job's managed identity.

Check which image tag the job is configured to use:

bash

az containerapp job show \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "properties.template.containers[0].image" \
  --output tsv

Verify that tag exists in ACR:

bash

az acr repository show-tags \
  --name ciriusagentsprod \
  --repository secops-agents \
  --orderby time_desc \
  --output table

If the configured tag is not in the list, the CI pipeline that builds the image may have failed.

Fix procedure:

Missing AcrPull role: Add it via Terraform. For immediate remediation:

bash

az role assignment create \
  --assignee <managed-identity-client-id> \
  --role "AcrPull" \
  --scope "/subscriptions/<logging-sub-id>/resourceGroups/rg-logging-logs/providers/Microsoft.ContainerRegistry/registries/ciriusagentsprod"

Tag not found: The CI build failed or was not run. Check GitHub Actions in bedrock-soc for a failed build step. Fix the build, merge to main, and the new image is pushed automatically.

Force re-pull without changing the image tag: Container Apps Jobs pull fresh on every execution. If you suspect a corrupted image layer, update the job to reference the same tag — Azure will re-pull:

bash

CURRENT_IMAGE=$(az containerapp job show --name job-orchestrator-cirius --resource-group rg-logging-logs --query "properties.template.containers[0].image" --output tsv)
az containerapp job update --name job-orchestrator-cirius --resource-group rg-logging-logs --image "$CURRENT_IMAGE"

Rollback to previous image tag:

bash

# Find the previous tag in ACR
az acr repository show-tags --name ciriusagentsprod --repository secops-agents --orderby time_desc --output table

# Roll the job back to that tag
az containerapp job update \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --image "ciriusagentsprod.azurecr.io/secops-agents:<previous-tag>"

Verify fix: Manually trigger the job and watch execution history. A successful image pull shows Running status within a few seconds of trigger; a pull failure shows Failed almost immediately.

3.7 Partial Run (Some Agents Succeed, Some Fail)

Symptoms:

Job shows Succeeded but fewer findings than usual are in SecOps
Some agent names are missing from the run logs
Findings timestamps show a gap for a specific agent's coverage area

Root cause investigation:

Orchestrator runs agents in sequence. An agent that raises an unhandled exception will be logged and skipped; the orchestrator continues to the next agent. Partial runs happen when individual agents error but do not crash the process.

Identify which agents ran vs. which did not by checking their log output:
kql
```
ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s == "job-orchestrator-cirius"
| where Log_s has "agent" or Log_s has "Agent"
| project TimeGenerated, Log_s
| sort by TimeGenerated asc
```
Look for agent start messages (e.g., Starting entra_agent) and completion messages. Agents that started but have no completion message before the next agent's start message errored and were skipped.

Check SecOps for which source types posted in the last cycle:

bash

curl -s -H "X-API-Key: $SECOPS_API_KEY" "https://soc.bedrockcybersecurity.org/api/incidents?created_since=5h&limit=100" | jq '[.[].agent_type] | unique'

Compare the list to the full expected set for that job.

For the specific agent that failed, find the traceback:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s == "job-orchestrator-cirius"
| where Log_s has "Traceback" or Log_s has "Exception"
| project TimeGenerated, Log_s
| sort by TimeGenerated asc

Recovery decision — re-run now vs. wait for next cycle:

Situation	Recommendation
Failed agents cover non-critical operational checks (backup status, cert expiry)	Wait for next scheduled cycle. Gap is acceptable.
Failed agents cover identity, network, or kill chain detection	Manually trigger the job immediately to close the gap.
SecurityAgent itself failed	All NEW findings are sitting untriaged. Trigger immediately.
All 3 jobs had partial runs simultaneously	Shared dependency issue (Key Vault, registry). Fix root cause first, then trigger all three.

Coverage impact of a missed cycle: Each missed 4-hour cycle means LAW events from that window are not checked against current known-good rules. Agents do not backfill past windows on the next run — they query from ago(4h). If a 4-hour window is fully missed, those events are only visible by running a manual KQL query against LAW directly. A missed cycle is a detection gap, not a data loss — the events still exist in LAW.

Verify fix: After re-triggering, check that all expected agent types appear in the SecOps incident feed within 30 minutes of trigger.

3.8 Memory Limit (OOM — Out of Memory)

Symptoms in logs / execution status:

Container OOMKilled
Job execution failed with exit code 137
MemoryError: unable to allocate

The job's container was killed by the OS because it exceeded its memory allocation.

Root cause investigation:

Check execution history for exit code 137 (Linux OOM kill signal):

bash

az containerapp job execution show \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --job-execution-name <execution-name> \
  --query "properties.status" \
  --output json

Identify which agent was running when the OOM occurred — the last log line before the process dies points to the agent:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s == "job-orchestrator-cirius"
| project TimeGenerated, Log_s
| sort by TimeGenerated desc
| take 20

The most common cause is a LAW query returning an unexpectedly large result set — for example, if a query that normally returns 500 rows returns 50,000 because of a burst in log volume or a missing time-range filter.

Fix procedure:

Immediate: Manually trigger the job — the offending condition may be transient.
If recurring: Find the agent pulling too much data and reduce the result set. Add a | take 10000 ceiling to the KQL query, or tighten the time filter from ago(4h) to a smaller window, or add a limit parameter to the API call.
Increase container memory: If the agents legitimately need more memory (e.g., kill chain agents processing high-volume EventID 4688 data):
bash
```
az containerapp job update \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --memory 2Gi \
  --cpu 1.0
```
Do this via Terraform for permanent changes.

Verify fix: On the next run, look for successful completion and absence of exit code 137 in execution history.

4. Manually Triggering a Job Run

Via Azure Portal

Portal → rg-logging-logs → [job name] → Run now

Via CLI

bash

# Trigger PROD job immediately
az containerapp job start \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs

# Trigger DDE job
az containerapp job start \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs

# Trigger AWS job
az containerapp job start \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs

When to Trigger Manually vs. Wait

Trigger immediately when:

A known outage caused missed detections (SecOps was down, Key Vault was unreachable)
You fixed a deployment bug and need to verify the fix works before the next scheduled run
An incident is active and you need fresh findings right now
A partial run missed identity, network, or kill chain agents

Wait for the next scheduled run when:

The failure was a transient external dependency (LAW throttle, OpenAI rate limit) and the next cycle is within 1–2 hours
The failure affected only non-critical operational agents (backup status, cert expiry)
You're in the middle of a fix that isn't deployed yet — triggering now just fails again

Running a Single Agent for Debugging

The orchestrator does not natively support running a single agent via CLI flags in the current architecture. To debug a specific agent in isolation:

Check whether the bedrock-soc repo has a --agent flag on orchestrator.py (look at orchestrator.py --help).

If yes:

bash

az containerapp job start \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --yaml-configuration '{"properties": {"template": {"containers": [{"args": ["--agent", "entra_agent"]}]}}}'

If the orchestrator does not support single-agent execution, run the agent locally using the bedrock-soc dev environment with a read-only test Key Vault. Do not run agents locally against production Key Vault or production SecOps.

5. Container Image Issues

Verify Current Image Tag Deployed

bash

az containerapp job show \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "properties.template.containers[0].image" \
  --output tsv

Do this for all three jobs if you suspect an image mismatch.

Check Available Tags in ACR

bash

# List all available tags, newest first
az acr repository show-tags \
  --name ciriusagentsprod \
  --repository secops-agents \
  --orderby time_desc \
  --output table

Verify an Image Tag Exists Before Deploying

bash

az acr repository show \
  --name ciriusagentsprod \
  --image secops-agents:<tag>

Returns image metadata if it exists; errors if it does not.

Force a Re-Pull Without Changing the Image Tag

Container Apps Jobs pull fresh on every execution — there is no caching between runs. To force a re-pull of the same tag (e.g., if you suspect a corrupted manifest):

bash

# Read the current image, then write it back — triggers a config update and forces re-evaluation
CURRENT_IMAGE=$(az containerapp job show --name job-orchestrator-cirius --resource-group rg-logging-logs --query "properties.template.containers[0].image" --output tsv)
az containerapp job update --name job-orchestrator-cirius --resource-group rg-logging-logs --image "$CURRENT_IMAGE"

Rollback to Previous Image Tag

bash

# Find previous tag
az acr repository show-tags --name ciriusagentsprod --repository secops-agents --orderby time_desc --output table

# Apply previous tag to a job
az containerapp job update \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --image "ciriusagentsprod.azurecr.io/secops-agents:<previous-tag>"

Do this for each affected job. Rollbacks via ad-hoc CLI will be overwritten on the next Terraform apply — file a Terraform PR to make the rollback permanent if needed.

6. SecOps API Health Check

Quick Health Check

bash

curl -s -o /dev/null -w "%{http_code}" https://secops.bedrockcybersecurity.org/health

Returns 200 if healthy. Any other response means the platform is degraded or down.

Check SecOps Container App Status

bash

az containerapp show \
  --name ca-secops-prod \
  --resource-group rg-logging-logs \
  --query "properties.{ProvisioningState:provisioningState,RunningStatus:latestRevisionFqdn}" \
  --output table

Check Active Revision and Running Replicas

bash

az containerapp revision list \
  --name ca-secops-prod \
  --resource-group rg-logging-logs \
  --query "[?properties.active==true].{Name:name,Replicas:properties.replicas,HealthState:properties.healthState,TrafficWeight:properties.trafficWeight}" \
  --output table

At least one replica should be Running. If replicas are at 0 or in a Degraded state, the platform is down.

Cold-Start Time Expectations

SecOps is a Container App (not a Job) — it is a persistent service. If it has been scaled down to 0 replicas due to inactivity, the first request after a cold start takes 30–60 seconds to serve while the container initializes. Agents may timeout during this window. If an agent run fails with a timeout error against secops.bedrockcybersecurity.org shortly after the SecOps app was restarted, wait 60 seconds and trigger the job again.

Database Connectivity Check

If the SecOps health endpoint returns 500 or 503, the platform may have lost database connectivity.

bash

# Check SecOps logs for DB connection errors
az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'ca-secops-prod' | where Log_s has 'psycopg' or Log_s has 'connection' or Log_s has 'database' | where TimeGenerated > ago(1h) | sort by TimeGenerated desc | take 20" \
  --output table

If you see PostgreSQL connection errors, check psql-secops-prod status in the portal. For deeper database operations, see PostgreSQL Operations.

7. LAW Query Debugging

Running Agent KQL Queries Directly in the Portal

Go to Portal → Log Analytics workspaces → cirius-logging-law-central → Logs
Paste the exact KQL the agent uses (find it in the agent source code in bedrock-soc)
Adjust the time range to match what the agent would query (typically ago(4h) or ago(6h))
Run and observe: row count, any errors, column names

This is the fastest way to determine whether an agent's data source is healthy independent of the agent itself.

CLI — Run a KQL Query Against LAW

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "<paste KQL here>" \
  --output table

Common KQL Errors Agents Encounter

Error	Cause	Fix
`Table X not found`	The table doesn't exist in this workspace yet — either the data source hasn't connected or no events have arrived since the workspace was created	Check that the DCR or connector pushing to this table is active
`Column 'X' does not exist`	A recent LAW schema change renamed a column	Update the agent's KQL to use the new column name; use `
`Query exceeded memory limit`	Time range too large or result set too big	Add `
`Request timeout`	Complex join or too much data	Simplify the query; use `
`Results are partial`	Query was truncated due to LAW limits	Use `

Check LAW Ingestion Lag

Events are not always available in LAW immediately. Ingestion lag is typically 2–5 minutes for most sources, but can reach 10–15 minutes for high-volume sources (Palo Alto CEF, SecurityEvent). If an agent queries ago(4h) and the job runs precisely at the 4-hour mark, some events from the last 5–10 minutes of the window may not be in LAW yet.

kql

// Check ingestion lag for a specific table
SecurityEvent
| where TimeGenerated > ago(1h)
| summarize IngestionDelay = avg(ingestion_time() - TimeGenerated)
| project IngestionDelayMinutes = IngestionDelay / 1m

If ingestion lag is above 15 minutes, investigate the DCR or connector feeding that table.

Check That a Table Exists and Has Recent Data

kql

// Check last event time for key tables
union
  (SecurityEvent | summarize LastEvent = max(TimeGenerated) | extend Table = "SecurityEvent"),
  (SignInLogs | summarize LastEvent = max(TimeGenerated) | extend Table = "SignInLogs"),
  (AuditLogs | summarize LastEvent = max(TimeGenerated) | extend Table = "AuditLogs"),
  (ContainerAppConsoleLogs_CL | summarize LastEvent = max(TimeGenerated) | extend Table = "ContainerAppConsoleLogs_CL")
| project Table, LastEvent
| sort by LastEvent asc

Any table with LastEvent more than 6 hours old (for high-frequency sources) may indicate a connector or DCR failure.

8. Partial Run Recovery

Determine Which Agents Ran and Which Did Not

Check SecOps for recent incident timestamps by agent type:

bash

curl -s -H "X-API-Key: $SECOPS_API_KEY" "https://soc.bedrockcybersecurity.org/api/incidents?created_since=5h&limit=200" | jq 'group_by(.agent_type) | map({agent: .[0].agent_type, count: length, latest: (map(.created_at) | max)})'

Cross-reference the agent list against the expected full set for the job. Any agent type with no findings is either silent (clean environment) or failed to run.

Distinguish silence from failure by checking logs:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(5h)
| where ContainerAppName_s == "job-orchestrator-cirius"
| where Log_s has "found 0 findings" or Log_s has "no findings" or Log_s has "Exception"
| project TimeGenerated, Log_s

"Found 0 findings" means the agent ran clean. An exception means it failed.

Re-run vs. Wait for Next Cycle

See the decision matrix in Section 3.7. Core rule: if any kill chain, identity, or network agent failed, trigger immediately. If only operational/inventory agents failed, wait.

How Missing a Cycle Affects Detection Coverage

Agents query LAW for the past 4 hours. On the next successful run, they query the fresh 4-hour window — they do not reach back to cover the missed window. A missed cycle creates a 4-hour blind spot in detection for that agent's domain.

During a known outage that spans one or more cycles, manually run the relevant KQL queries directly against LAW to cover the gap. The KQL Query Reference has the base queries for each detection category.

9. Email and Telegram Notification Failures

Architecture Note

bedrock-soc uses AWS SES via boto3 for email notifications — not ACS (ACS is used by bedrock-grc). Do not mix them. Any email debugging must go through SES, not the ACS email runner.

Heartbeat Email Not Arriving

The heartbeat email is sent after every job run. If it stops arriving:

Check job logs for the email send attempt:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s in ("job-orchestrator-cirius", "job-orchestrator-cirius", "job-orchestrator-cirius")
| where Log_s has "email" or Log_s has "notification" or Log_s has "heartbeat"
| project TimeGenerated, ContainerAppName_s, Log_s

If the job succeeded but no email log lines appear, the email step may have been skipped due to an earlier exception. Check if the orchestrator's email step is inside a try/except that swallows errors silently.

If the email was attempted but failed, look for SES errors:

ClientError: An error occurred (MessageRejected): Email address is not verified
ClientError: An error occurred (Throttling): Daily message quota exceeded

Common causes:

Symptom	Cause	Fix
`MessageRejected`	Sender address not verified in SES	Verify the sender in AWS SES → Verified identities in the logging account (038901680748)
`Throttling` / quota exceeded	SES sending quota hit	Check SES send statistics; request a quota increase if needed
Silent failure / no log line	Exception swallowed upstream in orchestrator	Fix error handling in orchestrator code
Email arrives but to wrong address	Recipient config wrong	Check the recipient list in Key Vault or environment config

Manual Notification Procedure

If automated notifications are broken and you need to send an immediate notification:

bash

# Send a manual heartbeat/status email via SES CLI
aws ses send-email \
  --from "secops@ciriusgroup.com" \
  --to "rgarshol@gmail.com" \
  --subject "Manual SecOps Status — $(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --text "Manual notification: automated heartbeat is not functioning. Job status: [insert status]. See runbook for next steps." \
  --region us-east-1 \
  --profile logging

Telegram Notification Failures

Telegram alerts fire only when SecurityAgent determines something is urgent. If Telegram alerts stop:

Check whether the SecurityAgent is running at all — if it failed, no Telegram messages are sent regardless of finding severity.

Check logs for Telegram API errors:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s in ("job-orchestrator-cirius", "job-orchestrator-cirius", "job-orchestrator-cirius")
| where Log_s has "telegram" or Log_s has "Telegram"
| project TimeGenerated, ContainerAppName_s, Log_s

Verify the Telegram bot token in Key Vault is still valid — tokens expire if the bot is revoked.

Note: Telegram is read-only for SecOps — buttons and interactive responses are not implemented. All actual alert responses and approvals go through the SecOps platform UI, not Telegram.

10. Escalation Path

What This Runbook Covers

This runbook covers operational troubleshooting: the agent pipeline, job execution, log reading, and platform health. It covers everything from "why didn't the job run" to "why are findings not reaching SecOps."

What Requires Rory's Involvement

Any code change to agent logic, orchestrator, or SecOps platform — these require a PR review
Adding or modifying known-good suppression rules — Rory is the final authority; nothing is suppressed without his approval
Secret rotation for any Key Vault secret
Increasing Container Apps Job CPU/memory limits (Terraform change)
Changing job schedules permanently (Terraform change)
Any architectural decision about the agent pipeline

When to Escalate to Arctic Wolf (MDR)

Arctic Wolf is the MDR and has full Azure access. Escalate to Arctic Wolf when:

You cannot determine whether a failure is technical or indicates active attacker activity
A break-glass account has been used (any of the 18 break-glass accounts — always CRITICAL, always escalate)
The detection pipeline has been down for more than one full cycle (4 hours) and you cannot restore it — Arctic Wolf can provide manual monitoring coverage during the outage
You are actively responding to an incident and need parallel monitoring while the pipeline is being restored

Do not wait to escalate if you suspect a security incident. Arctic Wolf acts first without needing approval.

Signs a Pipeline Failure Is a Security Incident (Not Just a Technical Fault)

Treat a failure as potentially security-related when any of the following are true:

Logs were cleared — event ID 1102 (Windows audit log cleared) or LAW data gap not explained by a known outage
All three jobs fail simultaneously at the same time — this is suspicious if no infrastructure change was made; legitimate technical faults typically affect one job or shared dependencies gradually, not all three at once at the same moment
Key Vault access is denied for the managed identity when the role assignment was not changed — an attacker who gained access may have modified Key Vault policies
Container image was replaced without a corresponding CI pipeline run — check ACR push history against GitHub Actions runs; unauthorized image pushes are a supply chain attack vector
The job ran but no findings exist in SecOps for an extended period, combined with known active threat indicators elsewhere in the environment

When in doubt, treat it as a security incident and engage Arctic Wolf. A false alarm costs an investigation. A missed incident costs far more.

11. Routine Monitoring Checklist

Daily

[ ] Check last execution status for all three jobs:

bash

for JOB in job-orchestrator-cirius job-orchestrator-cirius job-orchestrator-cirius; do echo "=== $JOB ===" && az containerapp job execution list --name $JOB --resource-group rg-logging-logs --query "sort_by([].{Start:properties.startTime,Status:properties.status}, &Start) | [-1]" --output table; done

[ ] Confirm heartbeat email arrived for each scheduled cycle
[ ] Check SecOps incident queue for any OPEN items older than 24 hours without an owner
[ ] Check Telegram for any urgent alerts

Weekly

[ ] Review execution history for patterns — any jobs that consistently run longer than others (possible resource pressure or growing data volume)?

bash

az containerapp job execution list --name job-orchestrator-cirius --resource-group rg-logging-logs --query "sort_by([].{Start:properties.startTime,Status:properties.status}, &Start) | [-30:]" --output table

[ ] Check error rate in logs over the past 7 days:

kql

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(7d)
| where ContainerAppName_s in ("job-orchestrator-cirius", "job-orchestrator-cirius", "job-orchestrator-cirius")
| extend IsError = Log_s has_any ("ERROR", "Exception", "Traceback", "CRITICAL")
| summarize TotalLines = count(), ErrorLines = countif(IsError == true) by ContainerAppName_s
| extend ErrorRate = round(100.0 * ErrorLines / TotalLines, 1)

[ ] Verify Key Vault secrets have not expired or are near expiration:

bash

az keyvault secret list --vault-name cirius-openai-kv-prod --query "[].{Name:name, Expires:attributes.expires, Enabled:attributes.enabled}" --output table

[ ] Review any known-good rules added in the past week — are they appropriately scoped?

Monthly

[ ] Verify all agents are producing findings — a "silent agent" (no findings in 30 days from an agent covering a non-empty environment) is either broken or was silently suppressed:
bash
```
curl -s -H "X-API-Key: $SECOPS_API_KEY" "https://soc.bedrockcybersecurity.org/api/incidents?created_since=30d&limit=1000" | jq '[.[].agent_type] | unique | sort'
```
Cross-reference against the full agent list. Any agent type not represented in the past 30 days warrants investigation.
[ ] Check LAW ingestion health for all key tables (see Section 7 — "Check That a Table Exists and Has Recent Data")
[ ] Review Container Apps Job resource utilization — if jobs are consistently using close to their CPU/memory limits, plan a capacity increase before they start OOMing:
bash
```
az containerapp job show --name job-orchestrator-cirius --resource-group rg-logging-logs --query "properties.template.containers[0].resources" --output json
```

[ ] Verify all three jobs are on the expected schedule:

bash

for JOB in job-orchestrator-cirius job-orchestrator-cirius job-orchestrator-cirius; do echo "$JOB:" && az containerapp job show --name $JOB --resource-group rg-logging-logs --query "properties.configuration.scheduleTriggerConfig.cronExpression" --output tsv; done

[ ] Confirm ACR image retention is not deleting tags that might be needed for rollback — keep at least the last 5 tags for each repository.

Container Apps Jobs Monitoring — execution history and log queries
Orchestrator Split Operations — 3-job architecture reference
SecOps Platform Guide — incident lifecycle and known-good rules
SecOps Platform Deployment — deploying new agent versions via CI
SecOps Findings Triage — working the open incident queue
Key Vault Secrets Workflow — secret rotation and management
KQL Query Reference — full LAW query library
Incident Response — confirmed incident playbook
Azure Daily Operations — Azure CLI auth and general commands
Known-Good Rule Management — suppression rule lifecycle

Document History

Date	Change	Author
May 2026	Initial draft — covers job execution history, log reading, 8 failure mode diagnoses, manual trigger, image management, SecOps health, LAW debugging, partial run recovery, email/Telegram notifications, escalation path, routine monitoring checklist.	Rory

SecOps Agent Pipeline — Troubleshooting Runbook ​

Architecture at a Glance ​

1. Finding Job Execution History ​

Azure Portal ​

CLI — List Recent Executions ​

CLI — Find the Last Failed Execution ​

CLI — Check All Three Jobs at Once ​

How to Read Execution Status ​

2. Reading Agent Logs ​

Portal — Log Stream (Live) ​

Log Analytics — General Log Query ​

Log Analytics — Logs for a Specific Execution ​

KQL — Agent Errors in the Last 24 Hours ​

KQL — Last Activity per Job (Detect Missing Runs) ​

UTC Time Correlation ​

3. Diagnosing Specific Failure Modes ​

3.1 LAW Query Timeout or Throttle (429 / HttpResponseError) ​

3.2 Azure OpenAI Rate Limit ​

3.3 SecOps API Unreachable ​

3.4 Key Vault Secret Fetch Failure ​

3.5 AWS Credential Expiration ​

3.6 Container Image Pull Failure ​

3.7 Partial Run (Some Agents Succeed, Some Fail) ​

3.8 Memory Limit (OOM — Out of Memory) ​

4. Manually Triggering a Job Run ​

Via Azure Portal ​

Via CLI ​

When to Trigger Manually vs. Wait ​

Running a Single Agent for Debugging ​

5. Container Image Issues ​

Verify Current Image Tag Deployed ​

Check Available Tags in ACR ​

Verify an Image Tag Exists Before Deploying ​

Force a Re-Pull Without Changing the Image Tag ​

Rollback to Previous Image Tag ​

6. SecOps API Health Check ​

Quick Health Check ​

Check SecOps Container App Status ​

Check Active Revision and Running Replicas ​

Cold-Start Time Expectations ​

Database Connectivity Check ​

7. LAW Query Debugging ​

Running Agent KQL Queries Directly in the Portal ​

CLI — Run a KQL Query Against LAW ​

Common KQL Errors Agents Encounter ​

Check LAW Ingestion Lag ​

Check That a Table Exists and Has Recent Data ​

8. Partial Run Recovery ​

Determine Which Agents Ran and Which Did Not ​

Re-run vs. Wait for Next Cycle ​

How Missing a Cycle Affects Detection Coverage ​

9. Email and Telegram Notification Failures ​

Architecture Note ​

Heartbeat Email Not Arriving ​

Manual Notification Procedure ​

Telegram Notification Failures ​

10. Escalation Path ​

What This Runbook Covers ​

What Requires Rory's Involvement ​

When to Escalate to Arctic Wolf (MDR) ​

Signs a Pipeline Failure Is a Security Incident (Not Just a Technical Fault) ​

11. Routine Monitoring Checklist ​

Daily ​

Weekly ​

Monthly ​

Related Documents ​

Document History ​

SecOps Agent Pipeline — Troubleshooting Runbook

Architecture at a Glance

1. Finding Job Execution History

Azure Portal

CLI — List Recent Executions

CLI — Find the Last Failed Execution

CLI — Check All Three Jobs at Once

How to Read Execution Status

2. Reading Agent Logs

Portal — Log Stream (Live)

Log Analytics — General Log Query

Log Analytics — Logs for a Specific Execution

KQL — Agent Errors in the Last 24 Hours

KQL — Last Activity per Job (Detect Missing Runs)

UTC Time Correlation

3. Diagnosing Specific Failure Modes

3.1 LAW Query Timeout or Throttle (429 / HttpResponseError)

3.2 Azure OpenAI Rate Limit

3.3 SecOps API Unreachable

3.4 Key Vault Secret Fetch Failure

3.5 AWS Credential Expiration

3.6 Container Image Pull Failure

3.7 Partial Run (Some Agents Succeed, Some Fail)

3.8 Memory Limit (OOM — Out of Memory)

4. Manually Triggering a Job Run

Via Azure Portal

Via CLI

When to Trigger Manually vs. Wait

Running a Single Agent for Debugging

5. Container Image Issues

Verify Current Image Tag Deployed

Check Available Tags in ACR

Verify an Image Tag Exists Before Deploying

Force a Re-Pull Without Changing the Image Tag

Rollback to Previous Image Tag

6. SecOps API Health Check

Quick Health Check

Check SecOps Container App Status

Check Active Revision and Running Replicas

Cold-Start Time Expectations

Database Connectivity Check

7. LAW Query Debugging

Running Agent KQL Queries Directly in the Portal

CLI — Run a KQL Query Against LAW

Common KQL Errors Agents Encounter

Check LAW Ingestion Lag

Check That a Table Exists and Has Recent Data

8. Partial Run Recovery

Determine Which Agents Ran and Which Did Not

Re-run vs. Wait for Next Cycle

How Missing a Cycle Affects Detection Coverage

9. Email and Telegram Notification Failures

Architecture Note

Heartbeat Email Not Arriving

Manual Notification Procedure

Telegram Notification Failures

10. Escalation Path

What This Runbook Covers

What Requires Rory's Involvement

When to Escalate to Arctic Wolf (MDR)

Signs a Pipeline Failure Is a Security Incident (Not Just a Technical Fault)

11. Routine Monitoring Checklist

Daily

Weekly

Monthly

Related Documents

Document History