Orchestrator Split Operations Guide

⚠️ Historical — superseded June 2026. The 3-job split described here shipped and was then consolidated into tenant-wide jobs: job-orchestrator-cirius (4h, PROD + DDE + AWS + workstations), job-mini-orchestrator-cirius (5 min rapid checks), job-secops-maintenance-cirius (daily posture). None of the job-secops-* / per-environment job names below exist anymore. For current operations use the Container Apps Jobs Monitoring runbook; source of truth is azure-infra/logging/secops_cirius.tf.

Status: Superseded — retained as historical/design reference
Registry: ciriusagentsprod.azurecr.io
Resource group: rg-logging-logs (logging subscription)
SecOps platform: secops.bedrockcybersecurity.org
LAW workspace: cirius-logging-law-central (ID 5d76d1f2)

1. Why the Split Happened

The Single-Job Problem

The original job-secops-prod ran all 17 agents in one Container Apps Job every 4 hours. All agents — PROD Azure, DDE Azure, and AWS — ran in the same execution context, posted to the same SecOps queue, and lived or died together.

Blast radius problem. A DDE agent hanging on a slow AVD API call, or an AWS agent hitting a throttle limit on a 7-account org sweep, delayed or killed PROD findings in the same execution. When the job container hit memory limits running 17 agents in sequence, PROD agents near the end of the queue got dropped first.

Masking. If a DDE agent threw an unhandled exception and the job exited early, PROD agents that hadn't run yet never ran. A DDE noise problem could leave PROD unmonitored for 4 hours without a clear signal — the job showed Failed but the reason was DDE, not PROD. With a shared job, a single Failed status gives no information about which environment is actually broken.

Identity and permission scope. One managed identity holding permissions to PROD Entra (tenant d477c9f8), DDE Entra (tenant ff1c5d68), and AWS across 7 accounts violates least privilege. The single identity was over-permissioned for any one task.

Schedule coupling. PROD is the highest-stakes environment and the target for 1-hour cadence once the agents are tuned. DDE and AWS don't need that frequency. With a single job, any schedule change applied to all 17 agents regardless of environment.

SecurityAgent context collapse. The LLM-based SecurityAgent receives a system prompt that frames its environment context before evaluating findings. A PROD SecurityAgent prompt (internal users, Entra, network) is meaningless when evaluating an AWS CloudTrail finding or a DDE AVD session-host event. One job meant one prompt for everything — either too broad to be useful or wrong for 2 of 3 environments.

What Each Job Now Owns

Job	Tenant / Cloud	Agents	Managed Identity Scope
`job-secops-prod`	Azure PROD (`d477c9f8`, `ciriusgroup.com`)	PROD Entra, Conditional Access, Activity Log, Defender for Cloud, network, operational, inventory	PROD tenant only
`job-secops-dde`	Azure DDE (`ff1c5d68`, `ciriusdde.com`)	DDE Entra, AVD session hosts, DDE Panorama, DDE AD, DDE Activity Log	DDE tenant only
`job-secops-aws`	AWS 7-account org	CloudTrail, Security Hub, IAM, GuardDuty, Config	AWS IAM role, no Azure permissions

All three jobs post findings to secops.bedrockcybersecurity.org. The source_system field on every finding distinguishes origin: PROD, DDE, or AWS. The SecOps platform treats them as separate environments in the incident dashboard.

2. Architecture Overview

Job Configuration Comparison

Attribute	`job-secops-prod`	`job-secops-dde`	`job-secops-aws`
Schedule	`0 /4 * ` (target: `0 * * *` once tuned)	`0 /4 * *`	`0 /4 * *`
Managed identity	PROD-scoped MI	DDE-scoped MI	None (AWS IAM via env vars / OIDC)
Azure tenant	`d477c9f8`	`ff1c5d68`	N/A
SecurityAgent prompt	PROD context (Entra users, internal network, Defender)	DDE context (AVD customers, HIPAA ePHI, Panorama)	AWS context (CloudTrail, multi-account, GuardDuty)
Analyst Agent	Yes — runs after SecurityAgent	Yes	Yes
Key Vault	`cirius-openai-kv-prod`	`cirius-openai-kv-prod`	`cirius-openai-kv-prod`
LAW queries	Queries `5d76d1f2` filtering to PROD tables	Queries `5d76d1f2` filtering to DDE tables	Does not query LAW — reads AWS APIs directly
`source_system` value	`PROD`	`DDE`	`AWS`

Job Isolation

Each job runs in its own execution context. A failure in job-secops-dde has no effect on job-secops-prod. The jobs do not share state, do not communicate with each other, and do not share a managed identity. The only shared resource is:

The SecOps platform at secops.bedrockcybersecurity.org (receives findings from all three)
The LAW workspace cirius-logging-law-central (queried by PROD and DDE jobs; not AWS)
The container registry ciriusagentsprod.azurecr.io (all three pull images from here)
Key Vault cirius-openai-kv-prod (stores OpenAI key and SecOps API key)

Agent Discovery

The orchestrator in each job discovers agents by scanning a directory at container startup. Agents are Python modules in a known subdirectory (e.g., agents/prod/, agents/dde/, agents/aws/). Each agent module exports a standard interface the orchestrator calls. This is how you add or remove agents without changing the orchestrator code — see Section 7.

3. Managing `job-secops-prod`

Check Execution History

bash

az containerapp job execution list \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --query "sort_by([].{Name:name,Start:properties.startTime,Status:properties.status}, &Start) | [-10:]" \
  --output table

A healthy PROD job shows Succeeded every 4 hours (eventually every 1 hour once tuned). Any gap longer than 5 hours (or 90 minutes after the cadence drops to 1 hour) means a missed run.

View Logs

bash

# All logs from PROD job in the last 6 hours
az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-secops-prod' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

For a specific failed execution:

bash

# Get the last failed execution's start time
az containerapp job execution list \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --query "[?properties.status=='Failed'] | [-1].{Name:name,Start:properties.startTime}" \
  --output json

Then query logs for that time window using the start time from the output above.

Manual Trigger

bash

az containerapp job start \
  --name job-secops-prod \
  --resource-group rg-logging-logs

Use after: fixing a deployment bug, recovering from an outage, or verifying a prompt change took effect.

PROD SecurityAgent Prompt

The PROD SecurityAgent prompt is stored in the bedrock-soc repo under agents/prod/ (exact file name: check the repo — look for security_agent.py or a prompts/ directory alongside PROD agents). The prompt frames the LLM with PROD-specific context before it evaluates each finding batch.

What the PROD prompt emphasizes:

Internal user population (ciriusgroup.com accounts)
Entra Conditional Access policy baseline — what deviations look like
Azure Activity Log: what constitutes anomalous management-plane activity
PROD network: Twingate as the normal remote access path — direct internet auth is suspicious
Defender for Cloud recommendations in the PROD tenant
Internal service accounts and their expected behavior
SOC2 compliance framing

To update the PROD prompt without a full redeploy:

If the prompt is loaded from a Key Vault secret or environment variable at job startup, update the secret and trigger a manual run:

bash

# Check how the prompt is delivered — look for env vars on the job
az containerapp job show \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --query "properties.template.containers[0].env" \
  --output table

If the prompt is embedded in the container image, a code change and redeploy is required (see Section 10). If it's an environment variable pointing to a Key Vault reference, update the secret value in Key Vault and start a new execution — no image rebuild needed.

PROD Agent Scope

PROD agents target the Azure PROD tenant (d477c9f8). They do not cross into the DDE tenant. Typical PROD findings: Conditional Access policy gaps, privileged role changes, unusual sign-in locations, Defender for Cloud high-severity recommendations, management plane operations from unexpected identities.

4. Managing `job-secops-dde`

Check Execution History

bash

az containerapp job execution list \
  --name job-secops-dde \
  --resource-group rg-logging-logs \
  --query "sort_by([].{Name:name,Start:properties.startTime,Status:properties.status}, &Start) | [-10:]" \
  --output table

View Logs

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-secops-dde' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

Manual Trigger

bash

az containerapp job start \
  --name job-secops-dde \
  --resource-group rg-logging-logs

What DDE Agents Look For vs PROD Agents

DDE is a customer-facing environment delivering Medicare billing via AVD. The risk profile is completely different from PROD:

Detection Category	DDE Focus	PROD Focus
Identity	DDE Entra (`ff1c5d68`) — external Medicare customer accounts, AVD access	PROD Entra (`d477c9f8`) — internal employees, admin accounts
Access method	AVD session hosts — browser/client access over internet; no Twingate	Twingate — VPN-style tunnel; direct internet access is suspicious
Compliance	HIPAA ePHI — Medicare data, session recording, data exfil risk	SOC2 — operational baseline
Network	DDE Panorama / Palo Alto firewall blocking/allow patterns, AVD session host traffic	PROD Palo Alto (no Panorama), Twingate access logs
AD	DDE on-prem AD (`ACTDIRAZC01`, `ACTDIRAZC02`) — group policy, domain controller events	PROD AD via Entra Connect from `ciriusgroup.internal`
High-value events	Customer account lockouts, ePHI application access outside business hours, AVD session host health degradation	Admin PIM activations, break-glass account use, resource deletions

DDE SecurityAgent prompt context:

AVD session host fleet as the delivery mechanism — host health matters
Medicare customers as users — account lockouts and failed logins may indicate credential stuffing
HIPAA ePHI context — any data-touching event gets higher severity
Panorama (DDE) as the firewall enforcement point — check rule changes and block logs
DDE AD as the identity backend — DC events, replication failures, group policy changes

DDE-Specific Failure Modes

The DDE managed identity must have permissions in the DDE tenant (ff1c5d68), not the PROD tenant. If someone assigns PROD tenant permissions to the DDE job's identity (easy mistake during setup), DDE agents silently get no data from DDE APIs and return empty findings. Check identity scope:

bash

az containerapp job show \
  --name job-secops-dde \
  --resource-group rg-logging-logs \
  --query "identity" \
  --output json

The client ID returned must be the DDE-scoped managed identity, not the PROD one.

5. Managing `job-secops-aws`

Check Execution History

bash

az containerapp job execution list \
  --name job-secops-aws \
  --resource-group rg-logging-logs \
  --query "sort_by([].{Name:name,Start:properties.startTime,Status:properties.status}, &Start) | [-10:]" \
  --output table

View Logs

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-secops-aws' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

Manual Trigger

bash

az containerapp job start \
  --name job-secops-aws \
  --resource-group rg-logging-logs

AWS-Specific Architecture

job-secops-aws does not use an Azure managed identity for its primary work. AWS credentials are delivered via environment variables or an OIDC federation that maps the Container Apps Job to an AWS IAM role. The job authenticates to AWS APIs directly.

Check how AWS credentials are configured:

bash

az containerapp job show \
  --name job-secops-aws \
  --resource-group rg-logging-logs \
  --query "properties.template.containers[0].env[?contains(name, 'AWS')].{Name:name,Value:value}" \
  --output table

The job still uses the SecOps API key from Key Vault (cirius-openai-kv-prod) to post findings — so it needs an Azure managed identity for Key Vault access only.

CloudTrail vs SecurityEvent

AWS agents read from CloudTrail (management events across all 7 accounts) and AWS Security Hub, not from the LAW workspace. PROD and DDE agents query LAW (cirius-logging-law-central) for SecurityEvent, SignInLogs, AuditLogs. AWS agents bypass LAW entirely:

Data Source	PROD / DDE	AWS
Sign-in events	LAW: `SignInLogs` table	CloudTrail: `ConsoleLogin` events
Privileged actions	LAW: `AuditLogs`	CloudTrail: management API calls
Security recommendations	LAW: `SecurityRecommendation`	AWS Security Hub findings
Network	LAW: `AzureFirewallNetworkRule`	VPC Flow Logs via CloudWatch or S3
Threat detection	Defender for Cloud → LAW	GuardDuty findings

AWS-Native Finding Types

AWS agents look for things specific to the AWS threat model:

IAM policy changes (especially wildcard grants or removing SCPs)
Root account activity — always critical
CloudTrail disabled or log validation failures in any account
GuardDuty findings not acknowledged within SLA
Public S3 bucket ACLs or misconfigurations
Cross-account role assumption from unexpected principals
Security Hub scores below threshold across any of the 7 accounts
EC2 instance metadata service (IMDSv1) access — should be IMDSv2-only
Unused IAM credentials older than 90 days

AWS Job Throttling

The AWS job queries across 7 accounts. CloudTrail and Security Hub have API rate limits. If the job runs too fast across too many accounts, it hits ThrottlingException. The orchestrator should have backoff logic, but check logs if you see intermittent failures:

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-secops-aws' | where Log_s contains 'Throttling' | where TimeGenerated > ago(24h) | sort by TimeGenerated desc | take 20" \
  --output table

6. Adjusting Schedules

How Container Apps Job Schedules Work

Schedules are set as CRON expressions on the job's trigger configuration. The format is standard 5-field CRON (minute hour day month weekday) in UTC.

View Current Schedule

bash

az containerapp job show \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --query "properties.configuration.scheduleTriggerConfig.cronExpression" \
  --output tsv

Change the Schedule

bash

az containerapp job update \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --cron-expression "0 * * * *"

0 * * * * = top of every hour. 0 */4 * * * = every 4 hours starting at 00:00 UTC.

Do this via Terraform, not ad-hoc CLI, for any permanent schedule change. Ad-hoc CLI changes will be overwritten on the next Terraform apply. Use CLI only for temporary changes during tuning — document what you changed and file a Terraform PR.

When to Change Schedules

Move PROD to 1-hour cadence when:

Agent runtime (job duration) is consistently under 30 minutes — check execution history
SecurityAgent is producing low false-positive rates (< 5% of findings are noise after 2 weeks)
No missed runs in the past 7 days
Key Vault and SecOps API latency is stable

Keep DDE and AWS at 4 hours unless:

A HIPAA audit or incident requires tighter monitoring — temporarily drop DDE to 1 hour
AWS accounts are showing elevated GuardDuty activity — drop AWS to 2 hours during incident

Stagger schedules to avoid LAW query contention. PROD and DDE both query LAW. If they run simultaneously, they compete for the same workspace. Consider offsetting: PROD at 0 */4 * * *, DDE at 30 */4 * * *.

7. Adding a New Agent to a Specific Job

How the Orchestrator Discovers Agents

The orchestrator scans a directory at container startup. Each agent is a Python module in a per-environment subdirectory. The orchestrator imports all modules it finds and calls the standard agent interface on each.

Typical structure in the bedrock-soc repo (verify against actual repo layout):

agents/
  prod/
    entra_agent.py
    conditional_access_agent.py
    network_agent.py
    ...
  dde/
    dde_entra_agent.py
    avd_agent.py
    panorama_agent.py
    ...
  aws/
    cloudtrail_agent.py
    guardduty_agent.py
    iam_agent.py
    ...
  shared/
    security_agent.py      # LLM-based, runs after domain agents
    analyst_agent.py       # Rules engine, runs after SecurityAgent

Each agent module must implement the interface the orchestrator expects — check an existing agent for the exact signature (typically an async def run(context) -> list[Finding] or equivalent).

Steps to Add a New Agent

1. Write the agent module. Place it in the correct subdirectory for the target job:

PROD agents → agents/prod/
DDE agents → agents/dde/
AWS agents → agents/aws/

Use an existing agent as a template. The agent must:

Return findings in the standard Finding schema (check models.py or equivalent)
Set source_system to the correct value (PROD, DDE, or AWS)
Handle its own exceptions — unhandled exceptions stop the orchestrator's run of that agent

2. Test locally. The repo should have a local dev mode. Run the agent in isolation before merging.

3. Open a PR to bedrock-soc. Normal CI runs (test → security-gate → build → deploy). The new agent is included in the image on merge to main.

4. The orchestrator picks it up automatically on the next execution — no orchestrator code change needed. Verify by checking the job logs after the next run and looking for the new agent's name in the startup output.

5. Monitor the first few runs. A new agent in production may produce noisy findings or hit permissions it doesn't have. Check logs after the first 2–3 executions:

bash

az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-secops-prod' | where Log_s contains 'your_new_agent' | where TimeGenerated > ago(12h) | sort by TimeGenerated asc" \
  --output table

Adding an Agent That Needs New Permissions

If the new agent calls an Azure API that the existing managed identity doesn't have access to, you need a Terraform change to add the role assignment before the agent can function. Add both the agent code and the Terraform role assignment in the same PR, or the agent will fail its first few runs with 403s.

For AWS agents: if the new agent needs access to a new AWS service (e.g., adding a Config agent), the IAM role used by job-secops-aws needs an updated policy. This is an AWS-side change, not Azure.

8. Cross-Environment Incidents

When an Incident Spans PROD and DDE

PROD and DDE are hard-isolated at the network and identity level. There is no trust between tenants, no network peering, and no shared credentials. However, the same threat actor can hold compromised credentials in both environments independently — particularly if a user with access to both tenants has their device compromised.

Scenario: A credential compromise where the attacker has access to both user@ciriusgroup.com (PROD) and their DDE AVD account (user@ciriusdde.com).

In this case, job-secops-prod and job-secops-dde will each independently generate findings for the same underlying incident, tagged source_system: PROD and source_system: DDE respectively.

Correlating Findings Across Jobs in SecOps

The SecOps platform does not automatically correlate findings across source_system values. You do it manually during triage.

KQL: Find findings from both PROD and DDE involving a specific user in the last 24h

Run this in SecOps's incident search or directly against the SecOps API:

bash

# Pull all incidents from the last 24h for both source systems
curl -s -H "X-API-Key: $SECOPS_API_KEY" \
  "https://soc.bedrockcybersecurity.org/api/incidents?created_since=24h&limit=100" \
  | jq '[.[] | select(.source_system == "PROD" or .source_system == "DDE")]'

Then look for overlapping usernames, timestamps, or IP addresses across the two result sets.

KQL: Look for the same IP appearing in both PROD and DDE sign-in logs

kql

let SuspiciousIPs =
  SignInLogs
  | where TimeGenerated > ago(24h)
  | where TenantId == "d477c9f8"  // PROD tenant
  | where ResultType != 0
  | distinct IPAddress;
SignInLogs
| where TimeGenerated > ago(24h)
| where TenantId == "ff1c5d68"  // DDE tenant
| where IPAddress in (SuspiciousIPs)
| project TimeGenerated, TenantId, UserPrincipalName, IPAddress, ResultType, ResultDescription, Location
| sort by TimeGenerated desc

KQL: Same user account pattern appearing in both tenants (different UPN domains)

kql

let ProdFailures =
  SignInLogs
  | where TimeGenerated > ago(6h)
  | where TenantId == "d477c9f8"
  | where ResultType != 0
  | extend BaseUser = tostring(split(UserPrincipalName, "@")[0])
  | distinct BaseUser;
SignInLogs
| where TimeGenerated > ago(6h)
| where TenantId == "ff1c5d68"
| extend BaseUser = tostring(split(UserPrincipalName, "@")[0])
| where BaseUser in (ProdFailures)
| project TimeGenerated, TenantId, UserPrincipalName, IPAddress, ResultType, Location
| sort by TimeGenerated desc

Cross-Environment Incident Response Steps

Identify the shared indicator (username pattern, IP, device ID, or timing correlation)
Open a single incident in the SecOps platform to track the cross-environment event — link the PROD and DDE incident IDs in the notes
Disable the account in PROD Entra (d477c9f8) and DDE Entra (ff1c5d68) independently — there is no cross-tenant disable; you must act in each tenant
Revoke sessions in both tenants:
- PROD: Entra Portal → User → Revoke sessions
- DDE: Entra Portal (DDE tenant) → User → Revoke sessions
Follow Incident Response for each environment separately — containment steps are tenant-specific

After containment, manually trigger both jobs to get fresh post-containment findings:

bash

az containerapp job start --name job-secops-prod --resource-group rg-logging-logs
az containerapp job start --name job-secops-dde --resource-group rg-logging-logs

9. Monitoring the Monitors

How to Know a Job Failed to Run

Method 1: Missed heartbeat alert. The SecOps platform tracks job heartbeats. If a job doesn't post any findings (including a heartbeat event) within its expected window, the platform sends a Telegram alert. A "No agent heartbeat from job-secops-prod" Telegram message is the primary alert path.

Method 2: Execution history gap. Check the last execution time against now:

bash

az containerapp job execution list \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --query "sort_by([].{Name:name,Start:properties.startTime,Status:properties.status}, &Start) | [-3:]" \
  --output table

If the most recent execution's Start is more than 5 hours ago (or more than 90 minutes once PROD moves to hourly), the job missed a run.

Method 3: KQL — check for execution gaps

kql

// Show last execution time for each job
ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(12h)
| where ContainerAppName_s in ("job-secops-prod", "job-secops-dde", "job-secops-aws")
| summarize LastLog = max(TimeGenerated) by ContainerAppName_s
| extend MinutesSinceLastLog = datetime_diff('minute', now(), LastLog)
| sort by MinutesSinceLastLog desc

If MinutesSinceLastLog exceeds the expected interval (240 for 4-hour jobs), that job has not logged anything recently.

KQL: Job Execution Health Dashboard

kql

// Execution status for all three jobs over the last 48 hours
ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(48h)
| where ContainerAppName_s in ("job-secops-prod", "job-secops-dde", "job-secops-aws")
| summarize
    TotalLogLines = count(),
    FirstLog = min(TimeGenerated),
    LastLog = max(TimeGenerated)
  by ContainerAppName_s, bin(TimeGenerated, 4h)
| sort by ContainerAppName_s asc, TimeGenerated asc

kql

// Error rate per job in the last 24 hours
ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(24h)
| where ContainerAppName_s in ("job-secops-prod", "job-secops-dde", "job-secops-aws")
| extend IsError = Log_s has_any ("ERROR", "Exception", "Traceback", "CRITICAL", "failed")
| summarize
    TotalLines = count(),
    ErrorLines = countif(IsError == true)
  by ContainerAppName_s
| extend ErrorRate = round(100.0 * ErrorLines / TotalLines, 1)
| sort by ErrorRate desc

kql

// Find agent-level failures (individual agents erroring within a successful job run)
ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(24h)
| where ContainerAppName_s in ("job-secops-prod", "job-secops-dde", "job-secops-aws")
| where Log_s has "Exception" or Log_s has "Traceback"
| project TimeGenerated, ContainerAppName_s, Log_s
| sort by TimeGenerated desc
| take 50

kql

// Verify findings are reaching SecOps — look for successful POST calls in agent logs
ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| where ContainerAppName_s in ("job-secops-prod", "job-secops-dde", "job-secops-aws")
| where Log_s has "posted" or Log_s has "finding" or Log_s has "201" or Log_s has "secops"
| project TimeGenerated, ContainerAppName_s, Log_s
| sort by TimeGenerated desc
| take 30

What to Do If a Job Stops Running

If the job shows Failed executions:

Get the error from the last failed execution's logs (see Section 3/4/5 per job)
Common causes: Key Vault access denied, managed identity misconfigured, image pull failure, Python import error in a new agent deployment

Fix the root cause, then manually trigger:

bash

az containerapp job start --name <job-name> --resource-group rg-logging-logs

If the job shows no executions at all (scheduler not firing):

Verify the job's schedule trigger is configured:

bash

az containerapp job show \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --query "properties.configuration.{TriggerType:triggerType,Cron:scheduleTriggerConfig.cronExpression}" \
  --output json

If TriggerType is not Schedule or Cron is missing/wrong, the trigger was misconfigured — update it:

bash

az containerapp job update \
  --name job-secops-prod \
  --resource-group rg-logging-logs \
  --cron-expression "0 */4 * * *"

Manually trigger to verify the job can execute before waiting for the next scheduled run

If the job shows Succeeded but no findings in SecOps:

This means the job ran but agents either found nothing or failed to post. Check:

bash

# Did agents find anything?
az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-secops-prod' | where Log_s has 'finding' or Log_s has 'Finding' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

# Did posting fail?
az monitor log-analytics query \
  --workspace 5d76d1f2 \
  --analytics-query "ContainerAppConsoleLogs_CL | where ContainerAppName_s == 'job-secops-prod' | where Log_s has 'secops' or Log_s has '422' or Log_s has '500' | where TimeGenerated > ago(6h) | sort by TimeGenerated asc" \
  --output table

If agents found nothing — that's valid if the environment was clean. If posting failed, check SecOps platform health and API key validity.

If all three jobs stop simultaneously:

The shared dependency is Key Vault (cirius-openai-kv-prod). All three jobs fetch the SecOps API key and OpenAI key from there. A Key Vault outage or firewall change blocking Container Apps egress to Key Vault will kill all three at once. Check Key Vault availability and the network security group / private endpoint config.

10. Tuning SecurityAgent Per Environment

What the SecurityAgent Does

The SecurityAgent is an LLM-based agent that runs after the domain agents in each job. It receives all findings from the current execution and applies judgment: Is this a real incident? What severity? What's the recommended action? It writes these decisions back to each finding before the Analyst Agent runs the rules engine on top.

The SecurityAgent uses Azure OpenAI (gpt-4o at cirius-openai-prod.openai.azure.com). Its system prompt is what makes it environment-aware. Without a tailored prompt, it would apply generic LLM security reasoning to every finding regardless of context.

PROD Prompt Emphasis

The PROD SecurityAgent prompt should emphasize:

User population: Internal ciriusgroup.com employees and service accounts — not customers, not contractors unless explicitly noted
Access baseline: Twingate is the normal remote path; direct internet-facing auth to PROD services is almost never expected
Privileged identity: PIM activations are expected but should be short-lived; standing privileged access is a finding
Change management: Terraform-driven infra changes post CM tickets; alert on changes that don't have a corresponding CM ticket
Break-glass: Any use of break-glass accounts is always CRITICAL regardless of context
Severity calibration: SOC2 framing; findings that affect the control environment get escalated severity

DDE Prompt Emphasis

The DDE SecurityAgent prompt should emphasize:

User population: Medicare customers accessing billing applications via AVD — external users who authenticate via DDE Entra (ff1c5d68)
HIPAA context: Any finding touching ePHI or ePHI-adjacent systems gets at minimum HIGH severity; data exfil patterns get CRITICAL
AVD as the delivery surface: Session host health failures, abnormal session durations, and off-hours access by customer accounts are meaningful signals
DDE Panorama / firewall: Policy changes or allow-list modifications on the DDE firewall are significant — DDE firewall is the ePHI perimeter
DDE AD: DC replication failures and domain controller events carry higher weight than in PROD because DDE AD is the identity backbone for customer-facing sessions
Account lockouts: Customer account lockouts may indicate credential stuffing against the Medicare billing portal — treat patterns of lockouts as HIGH, not LOW

AWS Prompt Emphasis

The AWS SecurityAgent prompt should emphasize:

Multi-account scope: Findings come from 7 accounts; include the account ID and account name in severity decisions — management/root account events are always elevated
Root activity: Any root account usage is CRITICAL, no exceptions
IAM blast radius: An IAM change in the management account can affect all 7 accounts; weight it accordingly
GuardDuty confidence: GuardDuty findings already include AWS's confidence score; the SecurityAgent should treat high-confidence GuardDuty findings as presumed-real unless there's a known-good match
CloudTrail integrity: CloudTrail being disabled or log file validation failing is a CRITICAL finding — it means the audit trail is broken
No Azure concepts: The prompt must not reference Entra, Conditional Access, Twingate, or any Azure identity concept — the LLM should reason only in AWS terms

How to Update Prompts Without a Full Redeploy

If prompts are stored as Key Vault secrets (preferred):

bash

# Check current secret value
az keyvault secret show \
  --vault-name cirius-openai-kv-prod \
  --name security-agent-prompt-prod \
  --query "value" \
  --output tsv

# Update the secret with a new prompt
az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name security-agent-prompt-prod \
  --value "$(cat new-prompt.txt)"

Then trigger a manual run to pick up the new prompt:

bash

az containerapp job start \
  --name job-secops-prod \
  --resource-group rg-logging-logs

No image rebuild. No deployment pipeline. The job reads the secret at startup.

If prompts are embedded in the container image:

A code change to the prompt file in bedrock-soc, followed by a merge to main, triggers the full CI pipeline (test → security-gate → build → deploy) and rebuilds the image. The new image is deployed to the Container Apps Job automatically. This takes longer (full CI run) and requires a PR.

Moving prompts from image-embedded to Key Vault secrets is the right architectural direction — it decouples prompt tuning from the release cycle.

Testing a prompt change before committing it:

Update the Key Vault secret with the candidate prompt
Manually trigger the job
Check the next execution's findings in SecOps for quality — are severities calibrated correctly? Is the SecurityAgent reasoning appropriately in its recommended_action field?
If the prompt is worse, revert the Key Vault secret to the previous value and trigger again

Tracking prompt versions: Key Vault maintains secret versions automatically. To see the history:

bash

az keyvault secret list-versions \
  --vault-name cirius-openai-kv-prod \
  --name security-agent-prompt-prod \
  --output table

To revert to a previous version:

bash

az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name security-agent-prompt-prod \
  --value "$(az keyvault secret show \
    --vault-name cirius-openai-kv-prod \
    --name security-agent-prompt-prod \
    --version <previous-version-id> \
    --query value -o tsv)"

Container Apps Jobs Monitoring — execution history, log queries, common failures
SecOps Platform Deployment — deploying new agent versions via CI
SecOps Platform Operations Guide — incident lifecycle, known-good rules, API
SecOps Findings Triage — working the open queue
DDE Environment Operations — DDE tenant reference
Incident Response — confirmed incident playbook
KQL Query Reference — additional LAW query patterns

Document History

Date	Change	Author
May 2026	Initial draft — covers split rationale, 3-job architecture, per-job operations, schedule management, agent addition, cross-environment incidents, job health monitoring, SecurityAgent prompt tuning.	Rory

Orchestrator Split Operations Guide ​

1. Why the Split Happened ​

The Single-Job Problem ​

What Each Job Now Owns ​

2. Architecture Overview ​

Job Configuration Comparison ​

Job Isolation ​

Agent Discovery ​

3. Managing job-secops-prod ​

Check Execution History ​

View Logs ​

Manual Trigger ​

PROD SecurityAgent Prompt ​

PROD Agent Scope ​

4. Managing job-secops-dde ​

Check Execution History ​

View Logs ​

Manual Trigger ​

What DDE Agents Look For vs PROD Agents ​

DDE-Specific Failure Modes ​

5. Managing job-secops-aws ​

Check Execution History ​

View Logs ​

Manual Trigger ​

AWS-Specific Architecture ​

CloudTrail vs SecurityEvent ​

AWS-Native Finding Types ​

AWS Job Throttling ​

6. Adjusting Schedules ​

How Container Apps Job Schedules Work ​

View Current Schedule ​

Change the Schedule ​

When to Change Schedules ​

7. Adding a New Agent to a Specific Job ​

How the Orchestrator Discovers Agents ​

Steps to Add a New Agent ​

Adding an Agent That Needs New Permissions ​

8. Cross-Environment Incidents ​

When an Incident Spans PROD and DDE ​

Correlating Findings Across Jobs in SecOps ​

Cross-Environment Incident Response Steps ​

9. Monitoring the Monitors ​

How to Know a Job Failed to Run ​

KQL: Job Execution Health Dashboard ​

What to Do If a Job Stops Running ​

10. Tuning SecurityAgent Per Environment ​

What the SecurityAgent Does ​

PROD Prompt Emphasis ​

DDE Prompt Emphasis ​

AWS Prompt Emphasis ​

How to Update Prompts Without a Full Redeploy ​

Related Documents ​

Document History ​

Orchestrator Split Operations Guide

1. Why the Split Happened

The Single-Job Problem

What Each Job Now Owns

2. Architecture Overview

Job Configuration Comparison

Job Isolation

Agent Discovery

3. Managing `job-secops-prod`

Check Execution History

View Logs

Manual Trigger

PROD SecurityAgent Prompt

PROD Agent Scope

4. Managing `job-secops-dde`

Check Execution History

View Logs

Manual Trigger

What DDE Agents Look For vs PROD Agents

DDE-Specific Failure Modes

5. Managing `job-secops-aws`

Check Execution History

View Logs

Manual Trigger

AWS-Specific Architecture

CloudTrail vs SecurityEvent

AWS-Native Finding Types

AWS Job Throttling

6. Adjusting Schedules

How Container Apps Job Schedules Work

View Current Schedule

Change the Schedule

When to Change Schedules

7. Adding a New Agent to a Specific Job

How the Orchestrator Discovers Agents

Steps to Add a New Agent

Adding an Agent That Needs New Permissions

8. Cross-Environment Incidents

When an Incident Spans PROD and DDE

Correlating Findings Across Jobs in SecOps

Cross-Environment Incident Response Steps

9. Monitoring the Monitors

How to Know a Job Failed to Run

KQL: Job Execution Health Dashboard

What to Do If a Job Stops Running

10. Tuning SecurityAgent Per Environment

What the SecurityAgent Does

PROD Prompt Emphasis

DDE Prompt Emphasis

AWS Prompt Emphasis

How to Update Prompts Without a Full Redeploy

Related Documents

Document History