Container Apps Jobs Monitoring

Overview

The Cirius Group security monitoring system runs as Azure Container Apps Jobs — not long-running services, but jobs that execute on a schedule and exit. Three jobs cover the three environments:

Job	Environment	Schedule	Agents
job-orchestrator-cirius	Azure PROD	Every 4 hours	Entra, network, operational, inventory agents
job-orchestrator-cirius	Azure DDE	Every 4 hours	DDE-specific agents
job-orchestrator-cirius	AWS all accounts	Every 4 hours	CloudTrail, Security Hub, IAM agents

All three jobs live in rg-logging-logs in the logging subscription.

When a job runs, it spawns all its agents, each agent posts findings to the SecOps platform, the SecurityAgent makes incident decisions, and the job exits. If the job fails or doesn't run, no findings are posted and alerts are not generated.

---

Checking Job Execution History

Via Azure Portal

Azure Portal → rg-logging-logs → [job name] → Execution history

Shows each execution with:

• Start time
• Duration
• Status: Succeeded / Failed / Running

A healthy job runs every 4 hours and shows Succeeded. Any Failed status needs investigation. If there's a gap in the execution history longer than 5 hours, the job missed a run.

Via CLI

# List recent executions for a job
az containerapp job execution list \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --output table \
  --query "sort_by([].{Name:name,StartTime:properties.startTime,Status:properties.status}, &StartTime) | [-10:]"

Check the last run time — should be within the last 4 hours.

---

Reading Job Logs

Finding the Logs for a Specific Execution

Each job execution has its own replica. Logs from a completed execution are in Log Analytics:

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-orchestrator-cirius' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

Replace job-orchestrator-cirius with job-orchestrator-cirius or job-orchestrator-cirius as needed.

Reading a Failed Execution's Logs

# Get the name of the most recent failed execution
az containerapp job execution list \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs \
  --query "[?properties.status=='Failed'] | [-1].name" \
  --output tsv

Then query logs for that execution:

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-orchestrator-cirius' | where TimeGenerated between (datetime('START') .. datetime('END')) | sort by TimeGenerated asc" \
  --output table

Replace START and END with the execution's start/end times.

Via Portal (Log Stream)

For a currently running job: Azure Portal → rg-logging-logs → [job name] → Log stream

Shows live stdout. Only works while the execution is in progress.

---

Manually Triggering a Job

To run a job immediately outside its schedule:

az containerapp job start \
  --name job-orchestrator-cirius \
  --resource-group rg-logging-logs

This starts a new execution immediately. Monitor it in the execution history or via log stream.

When to use:

• Testing a newly deployed agent fix
• After a known outage — catch up on missed detections
• Verifying the job is healthy after infrastructure changes

---

Common Failure Patterns

Key Vault Access Denied

The job's managed identity must have Key Vault Secrets User on cirius-openai-kv-prod. If the role was accidentally removed, all agents fail at startup.

Error in logs: AzureKeyVaultSecretsMissingException or 403 Forbidden from Key Vault

Fix:

az role assignment create \
  --assignee <managed-identity-client-id> \
  --role "Key Vault Secrets User" \
  --scope /subscriptions/.../resourceGroups/rg-logging-logs/providers/Microsoft.KeyVault/vaults/cirius-openai-kv-prod

SecOps API Unreachable

The job posts findings to https://secops.bedrockcybersecurity.org. If the SecOps platform is down, agents fail to post and the job exits with an error.

Error in logs: ConnectionRefused or 502 from secops.bedrockcybersecurity.org

Fix: Restore the SecOps platform first (see SecOps Platform Deployment). Then manually trigger the job to catch up.

Agents are configured to fail-open — they log the failure and continue to the next agent rather than stopping the whole job on a single agent's error.

Azure Credential / OIDC Failure

The job uses managed identity to authenticate to Azure APIs (Entra, Activity Log, Security Center). If the managed identity has been removed or its role assignments changed, Azure calls fail.

Error in logs: AuthenticationError or 403 from management.azure.com

Fix: Verify managed identity role assignments match what Terraform defines:

az role assignment list --assignee <managed-identity-client-id> --output table

If roles are missing, reapply via Terraform (add to the next infra PR).

Container Image Pull Failure

If the job can't pull its image from ciriusagentsprod.azurecr.io:

Error in logs: Failed to pull image or unauthorized: authentication required

Fix: Verify the Container Apps Job has AcrPull role on the container registry:

az role assignment list \
  --scope /subscriptions/.../providers/Microsoft.ContainerRegistry/registries/ciriusagentsprod \
  --output table | grep AcrPull

Python Runtime Errors

Individual agent errors (not whole-job failures) show up as tracebacks in the logs. The job continues to the next agent. These are bugs in agent code — fix in bedrock-soc repo and redeploy.

---

Verifying Findings Are Posting

After a job runs, verify findings reached the SecOps platform:

# Check for findings created in the last 6 hours
curl -s -H "X-API-Key: $SECOPS_API_KEY" \
  "https://soc.bedrockcybersecurity.org/api/incidents?created_since=6h&limit=10" | jq '.'

Or in the SecOps UI: Incidents → filter by created date → last 4 hours

If the job succeeded but no findings were posted, the agents may have found nothing (valid if the environment is clean) or the SecOps API endpoint for findings is erroring silently. Check SecOps logs:

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'ca-soc-prod' | where Log_s contains 'ingest' | where TimeGenerated > ago(6h) | sort by TimeGenerated desc | take 20" \
  --output table

---

Alerting on Job Failures

The SecOps platform sends a Telegram alert if no heartbeat has been received in the expected window. If you receive a "No agent heartbeat" alert:

1. Check the job execution history (was the job supposed to run?)
2. Check for failed executions
3. Read the logs for the failed execution
4. Fix the root cause and manually trigger a job run

If the Telegram alert itself stopped arriving, check the SecOps notification agent is running and the Telegram bot token is valid.

---

Related Documents

• SecOps Platform Deployment — deploying new agent versions
• SecOps Platform Operations Guide — the web platform that receives findings
• SecOps Findings Triage — what to do with the findings after they post
• Azure Daily Operations — Azure CLI auth and general commands

---

Document History

Date	Change	Author
May 2026	Initial draft — job execution history, log queries, manual trigger, common failure patterns (Key Vault, SecOps API, managed identity, image pull, Python errors), findings verification, failure alerting.	Rory