Skip to content

SecOps Change Management Operations Guide

System: SecOps Platform — secops.bedrockcybersecurity.org
Database: psql-secops-prod (PostgreSQL)
Final Authority: Rory — no CM ticket auto-approves without his PR review
Related doc: cicd/pipeline-overview.md — CI/CD pipeline mechanics


1. What CM Tickets Are and How They Flow

A change management (CM) ticket is a time-stamped record in the SecOps platform that says "a planned change is happening in this environment right now." The SecurityAgent uses this record to decide whether a security alert is a false positive triggered by the change, or a real incident that needs investigation.

The Full Flow: PR Merge → SecurityAgent Suppression

Developer opens PR


Checkov + Trivy scan (Phase 1 gate — must pass before merge)


Rory reviews diff and plan  ←── Final authority, no auto-approve


PR merged to main


GitHub Actions: ingest-changes.yml fires (post-merge trigger)
        │   POST /api/changes with title, description, repo, pr_number,
        │   merged_by, environment (derived from repo name)

SecOps platform creates CM ticket — stored in psql-secops-prod


Terraform apply runs (auto, -auto-approve, OIDC auth)
        │   Infrastructure changes hit production

Security agents fire next cycle (up to 4 hours later)
        │   Agent detects anomaly (new resource, auth event, config change)

SecurityAgent calls GET /api/changes/recent
        │   Returns all CM tickets created in the last 2 hours

Environment match found?
  YES → Incident suppressed; moved to CLOSED with CM reference logged
  NO  → Incident raised as NEW → OPEN → analyst triage

Environment Mapping (Repo → CM Ticket Environment)

The ingest-changes.yml workflow sets the environment field based on which repository fired the workflow:

RepositoryCM Environment
azure-infraPROD
aws-infraAWS
azure-dde-infraDDE
ops-automationALL
bedrock-docsALL

ALL means the CM ticket suppresses alerts across all environments. Use it only for changes that genuinely affect all environments (e.g., agent logic changes in ops-automation).

CM Ticket Fields

FieldTypeDescription
titlestringPR title or manual change title
descriptionstringPR body summary or manual change description
repostringSource repository name
pr_numberintegerGitHub PR number (null for manual tickets)
merged_bystringGitHub username of the person who merged
environmentenumPROD, DDE, AWS, or ALL
created_attimestampWhen the ticket was created (UTC)

2. The Suppression Window: Exactly How It Works

What SecurityAgent Does

On every agent cycle, SecurityAgent calls GET /api/changes/recent before deciding whether to raise an incident. This endpoint returns all CM tickets created in the last 2 hours (server-side filter — the window is not configurable per call).

GET https://soc.bedrockcybersecurity.org/api/changes/recent
X-API-Key: <agent key from Key Vault>

Response:

json
[
  {
    "id": 142,
    "title": "Add NSG rule for Velociraptor collector",
    "repo": "azure-infra",
    "pr_number": 87,
    "merged_by": "rory",
    "environment": "PROD",
    "created_at": "2026-05-25T14:03:22Z"
  }
]

What "Matching" Means

An open CM ticket suppresses an incident when both conditions are met:

  1. Environment match — the incident's source_system maps to the CM ticket's environment:

    • Source system PROD → matches CM tickets with environment = PROD or environment = ALL
    • Source system DDE → matches CM tickets with environment = DDE or environment = ALL
    • Source system AWS → matches CM tickets with environment = AWS or environment = ALL
    • Any source_system → always matches CM tickets with environment = ALL
  2. Time window — the CM ticket's created_at is within 2 hours of the current time.

There is no additional field matching — SecurityAgent does not match on resource name, IP address, or specific event type. If a CM ticket is open for the right environment within the window, all alerts from that environment are suppressed for the duration of the window.

This is intentional: a Terraform apply can generate dozens of different alert types (new resource creations, auth events, configuration changes). Matching on environment + time is simpler and less fragile than trying to match each alert to its specific change.

What Happens When Suppression Fires

When SecurityAgent determines an incident matches an open CM ticket:

  1. The incident is created in the database with status CLOSED (it is recorded — not dropped).
  2. The closed_reason field is set to change-management.
  3. The cm_ticket_id field is populated with the suppressing ticket's ID.
  4. No Telegram notification fires.
  5. The incident is visible in the SecOps UI under the CLOSED filter, with the CM reference.

The finding is recorded, not discarded. You can always query what was suppressed and which CM ticket suppressed it (see Section 8 for audit queries).

Fail-Open: When SecOps Is Unreachable

If GET /api/changes/recent times out or returns a non-200 response, SecurityAgent fails open: it raises the incident as NEW and skips suppression. No CM ticket data means no suppression. This is by design — losing suppression context is recoverable; losing a real incident is not.


3. Viewing CM Tickets

SecOps Platform UI

  1. Go to https://secops.bedrockcybersecurity.org — authenticate with your ciriusgroup.com account (MFA required).
  2. Navigate to Change Management in the left sidebar.
  3. The default view shows recent tickets. Filter by environment using the dropdown.
  4. Click any ticket to see full details including the PR link (if automated).

API Queries

Recent tickets (last 2 hours — what SecurityAgent sees):

bash
curl -s https://soc.bedrockcybersecurity.org/api/changes/recent \
  -H "X-API-Key: $(az keyvault secret show --vault-name cirius-openai-kv-prod --name secops-api-key --query value -o tsv)" \
  | jq '.[] | {id, title, environment, created_at, merged_by}'

All tickets today:

bash
curl -s "https://soc.bedrockcybersecurity.org/api/changes?since=$(date -u +%Y-%m-%dT00:00:00Z)" \
  -H "X-API-Key: <key>" \
  | jq '.[] | {id, title, repo, environment, created_at}'

Tickets for a specific environment:

bash
curl -s "https://soc.bedrockcybersecurity.org/api/changes?environment=PROD&limit=20" \
  -H "X-API-Key: <key>" \
  | jq .

Look up a specific ticket by ID:

bash
curl -s https://soc.bedrockcybersecurity.org/api/changes/142 \
  -H "X-API-Key: <key>" \
  | jq .

4. Manual CM Ticket Creation

When to Create One Manually

Normal deployments go through the PR flow and get CM tickets automatically. Create a manual ticket when:

  • Emergency change — production is breaking and you need to make a change that cannot wait for a PR (e.g., reverting a bad NSG rule directly via Azure CLI with Rory on the call).
  • Maintenance window work — you are running a manual config change, certificate rotation, or DB update outside the normal CI/CD path.
  • Manual config change — adjusting a Palo Alto rule via Panorama, running a manual Key Vault rotation, or making a change through the Azure Portal that Terraform does not manage yet.
  • Ingest failure recovery — the ingest-changes.yml workflow failed and SecurityAgent never got the CM ticket for a real PR merge (see Section 6).

Create the manual ticket before making the change, not after. The suppression window starts at ticket creation time. If you create it after making the change, you have already missed the window.

How to POST a Manual Ticket

Via curl:

bash
curl -s -X POST https://soc.bedrockcybersecurity.org/api/changes \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <key>" \
  -d '{
    "title": "Emergency NSG rule revert — azure-infra",
    "description": "Reverting NSG rule that blocked Velociraptor collector. Applied directly via Azure CLI. PR to follow.",
    "repo": "azure-infra",
    "pr_number": null,
    "merged_by": "rory",
    "environment": "PROD"
  }' | jq .

Via Python (for scripts):

python
import requests
import os

SECOPS_URL = "https://secops.bedrockcybersecurity.org"
API_KEY = os.environ["SECOPS_API_KEY"]

payload = {
    "title": "Manual cert rotation — cirius-openai-kv-prod",
    "description": "Rotating TLS cert for secops.bedrockcybersecurity.org. Performed via Key Vault portal. No Terraform change.",
    "repo": None,
    "pr_number": None,
    "merged_by": "rory",
    "environment": "PROD",
}

resp = requests.post(
    f"{SECOPS_URL}/api/changes",
    json=payload,
    headers={"X-API-Key": API_KEY},
    timeout=10,
)
resp.raise_for_status()
print(resp.json())

Required Fields

All fields are required for automated tickets. For manual tickets, repo and pr_number may be null.

FieldManual RequiredNotes
titleYesDescribe the change concisely
descriptionYesWhat changed, why, who authorized it
repoNo (null ok)Set to repo name if related to a repo
pr_numberNo (null ok)Set to PR number if related to a PR
merged_byYesYour identity — must be accurate for audit
environmentYesPROD, DDE, AWS, or ALL

5. Emergency Changes

An emergency change is any change to production that cannot go through the normal PR flow because waiting for CI/CD would cause or extend an outage.

Procedure

Before touching anything:

  1. Create a CM ticket:

    bash
    curl -s -X POST https://soc.bedrockcybersecurity.org/api/changes \
      -H "Content-Type: application/json" \
      -H "X-API-Key: <key>" \
      -d '{
        "title": "EMERGENCY: <one-line description>",
        "description": "Production issue: <what is broken>. Emergency fix: <what you are doing>. Authorized by Rory.",
        "repo": "<affected repo or null>",
        "pr_number": null,
        "merged_by": "rory",
        "environment": "PROD"
      }' | jq '{id, created_at}'

    Note the ticket ID. The 2-hour suppression window starts now.

  2. Confirm the ticket is visible:

    bash
    curl -s https://soc.bedrockcybersecurity.org/api/changes/recent \
      -H "X-API-Key: <key>" \
      | jq '.[] | select(.title | startswith("EMERGENCY"))'

Make the change — apply the fix directly via Azure CLI, portal, or Terraform CLI with Rory's explicit authorization on the call.

After the change:

  1. Verify the fix worked. Check the system, confirm the outage condition is resolved.

  2. Document in GitHub — open a follow-up PR within 24 hours that captures the change in Terraform (if applicable). Reference the CM ticket ID in the PR description.

  3. If the emergency is in a Terraform-managed resource, the follow-up PR must import the current resource state so Terraform doesn't revert the fix on the next apply.

Emergency Time Limit

The suppression window is 2 hours from ticket creation. If the emergency extends beyond 2 hours, create a maintenance window instead (Section 9) — it has an explicit end time and suppresses all alerts for the duration.


6. Failed CM Ingest

What Failure Looks Like

If the ingest-changes.yml GitHub Actions workflow fails after a PR merge:

  • A real PR was merged and Terraform applied successfully.
  • No CM ticket was created in SecOps.
  • When agents run their next cycle (within 4 hours), they will detect infrastructure changes.
  • GET /api/changes/recent returns no matching ticket.
  • SecurityAgent raises incidents for legitimate change activity.
  • You get alerts (possibly Telegram if CRITICAL/HIGH) for a deployment you authorized.

How to Detect a Failed Ingest

Step 1 — Check the GitHub Actions run:

Go to the repository on GitHub → Actions tab → filter by ingest-changes.yml (or search "ingest"). Find the workflow run triggered by the PR merge. If it shows a red ✗ or yellow warning, the ingest failed.

Alternatively, via CLI:

bash
gh run list --repo Cirius-Group-Inc/<repo-name> --workflow ingest-changes.yml --limit 5

Step 2 — Check SecOps for the expected ticket:

bash
curl -s "https://soc.bedrockcybersecurity.org/api/changes?limit=10" \
  -H "X-API-Key: <key>" \
  | jq '.[] | {id, title, repo, pr_number, created_at}' | head -40

If the PR merge is not in the list, ingest failed.

Recovery: Create the Missing Ticket Manually

Create the manual ticket referencing the actual PR:

bash
curl -s -X POST https://soc.bedrockcybersecurity.org/api/changes \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <key>" \
  -d '{
    "title": "PR #<number>: <PR title> [manual re-ingest]",
    "description": "Ingest workflow failed for PR #<number>. Ticket created manually to restore CM coverage. PR: https://github.com/Cirius-Group-Inc/<repo>/pull/<number>",
    "repo": "<repo-name>",
    "pr_number": <number>,
    "merged_by": "<github username of merger>",
    "environment": "<PROD|AWS|DDE|ALL>"
  }' | jq .

If agents have already fired and created false-positive incidents, close them with a note referencing the CM ticket ID:

bash
# Find incidents raised during the deployment window
curl -s "https://soc.bedrockcybersecurity.org/api/incidents?status=NEW&since=<deploy-time-ISO>" \
  -H "X-API-Key: <key>" \
  | jq '.[] | {id, title, source_system, created_at}'

# Close each false positive with CM reference
curl -s -X PATCH https://soc.bedrockcybersecurity.org/api/incidents/<incident-id> \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <key>" \
  -d '{
    "status": "CLOSED",
    "resolution_note": "False positive — legitimate PR #<number> deployment. CM ticket <ticket-id> created after ingest failure.",
    "cm_ticket_id": <ticket-id>
  }' | jq .

Investigating the Workflow Failure

Common causes:

  • API key expired or rotated — check Key Vault secret secops-api-key; confirm the GitHub Actions secret matches.
  • SecOps platform down — if SecOps was unreachable at merge time, the POST failed. Restore SecOps first, then create the missing ticket.
  • Network/timeout — transient. Retry by triggering the workflow manually if it supports re-run, or just create the ticket manually.
  • Workflow YAML syntax error — check the Actions log for parse errors. Fix in a follow-up PR.

7. CM Tickets vs. Known-Good Rules: When to Use Which

These are two separate suppression mechanisms with different purposes. Understanding the difference prevents misuse.

CM TicketsKnown-Good Rules
Duration2 hours from creationIndefinite (until disabled or expired)
ScopeAll alerts from matching environmentSpecific field match (actor, IP, event type)
Approval requiredNo (auto-created from PR, or self-service manual)Yes — Rory must explicitly approve
Audit trailYes — every CM ticket recorded with PR referenceYes — hit counts, creation/approval timestamps
Use casePlanned change in progressRecurring benign pattern that will never be a real threat

When to Use CM Tickets

  • You are deploying infrastructure changes (PR flow or manual).
  • You are running a maintenance window.
  • You need to suppress alerts for a bounded time period around a specific change.
  • The suppression need is temporary — once the change is stable, alerts should resume normally.

When to Use Known-Good Rules

  • A specific automated behavior fires repeatedly across many agent cycles.
  • The pattern is verified benign (not just "this time").
  • The pattern will continue indefinitely (not just during a deployment).
  • Examples: scheduled scan from a known IP, MDM bulk check-in event type, service account token refresh.

What Not to Do

Do not create a known-good rule to silence alerts that should be covered by CM tickets. If Terraform deployments consistently generate noisy alerts, the right fix is ensuring CM tickets are created (automated via PR flow) and that the ingest workflow is reliable — not blanket suppression of all deployment-related events via a known-good rule.

Do not rely on CM tickets for permanently recurring patterns. A 2-hour CM ticket opened before every scheduled maintenance task is operational overhead. If the maintenance is on a fixed schedule and generates the same alerts every time, evaluate whether a known-good rule (with Rory approval) is more appropriate.


8. Audit Trail: HIPAA, SOC 2, and CM History

HIPAA Requirement

Change management is a required control under HIPAA §164.308(a)(5)(ii)(B) — Security Awareness and Training, and more broadly under the Security Rule's requirement to implement policies and procedures for guarding against, detecting, and reporting malicious software and unauthorized access. A documented, time-stamped record of every infrastructure change — who made it, what was changed, when — is part of this control.

Every CM ticket (automated or manual) records:

  • The change title and description
  • The source repository and PR number
  • The identity of the person who merged or created the ticket
  • The environment affected
  • The timestamp (UTC, stored in PostgreSQL)

This record is stored in psql-secops-prod and is not deletable through the API. The database has WORM-equivalent retention controls in place.

SOC 2 CC8.1 (Change Management)

SOC 2 CC8.1 requires that changes to infrastructure are authorized, tested, and documented before implementation. CM tickets provide the implementation layer of this control:

  • Authorized: Rory's PR review is the authorization gate. No PR merges without his approval. The merged_by field proves who authorized execution.
  • Documented: CM ticket captures the PR title, description, and repository. The full diff is in GitHub history.
  • Traceable: CM ticket ID links the SecOps suppression record to the GitHub PR.

For SOC 2 evidence collection, export the CM ticket history for the audit period:

bash
curl -s "https://soc.bedrockcybersecurity.org/api/changes?since=2026-01-01T00:00:00Z&until=2026-03-31T23:59:59Z&limit=1000" \
  -H "X-API-Key: <key>" \
  | jq '[.[] | {id, title, repo, pr_number, merged_by, environment, created_at}]' \
  > cm-tickets-q1-2026.json

PostgreSQL Queries for CM History

Direct database access requires the psql-secops-prod connection string from Key Vault.

All CM tickets in a date range:

sql
SELECT
  id,
  title,
  repo,
  pr_number,
  merged_by,
  environment,
  created_at
FROM change_tickets
WHERE created_at BETWEEN '2026-01-01' AND '2026-03-31 23:59:59'
ORDER BY created_at DESC;

Count tickets per repository (deployment frequency):

sql
SELECT
  repo,
  COUNT(*) AS ticket_count,
  MIN(created_at) AS first_change,
  MAX(created_at) AS last_change
FROM change_tickets
WHERE created_at > NOW() - INTERVAL '90 days'
GROUP BY repo
ORDER BY ticket_count DESC;

All incidents suppressed by CM tickets (suppression audit):

sql
SELECT
  i.id AS incident_id,
  i.title AS incident_title,
  i.source_system,
  i.severity,
  i.created_at AS incident_time,
  i.closed_at,
  ct.id AS cm_ticket_id,
  ct.title AS cm_title,
  ct.repo,
  ct.pr_number,
  ct.merged_by
FROM incidents i
JOIN change_tickets ct ON i.cm_ticket_id = ct.id
WHERE i.closed_reason = 'change-management'
ORDER BY i.created_at DESC;

CM tickets with no corresponding suppressed incidents (changes that did not trigger alerts):

sql
SELECT
  ct.id,
  ct.title,
  ct.repo,
  ct.environment,
  ct.created_at
FROM change_tickets ct
LEFT JOIN incidents i ON i.cm_ticket_id = ct.id
WHERE i.id IS NULL
  AND ct.created_at > NOW() - INTERVAL '30 days'
ORDER BY ct.created_at DESC;

Manual tickets vs. automated tickets:

sql
SELECT
  CASE WHEN pr_number IS NULL THEN 'manual' ELSE 'automated' END AS ticket_source,
  COUNT(*) AS count,
  environment
FROM change_tickets
WHERE created_at > NOW() - INTERVAL '90 days'
GROUP BY ticket_source, environment
ORDER BY ticket_source, environment;

9. Maintenance Windows

A maintenance window is a broader suppression mechanism that suspends all SecurityAgent incident creation for a defined time period. It is distinct from a CM ticket — a CM ticket suppresses based on environment match within a rolling 2-hour window; a maintenance window suppresses everything from all environments until the window expires.

When to Use a Maintenance Window

Appropriate:

  • Bulk Terraform deployments across multiple repos where you expect significant alert volume.
  • Firewall rule changes that will trigger network anomaly alerts.
  • Server patching or reboots across multiple systems.
  • Large-scale infrastructure migrations where the environment will be legitimately noisy.

Not appropriate:

  • Routine single-PR deployments — use CM tickets (auto-created from ingest-changes.yml).
  • Hiding alerts from a real incident under investigation.
  • Suppressing alerts because you expect to fail a compliance check.

Creating a Maintenance Window

bash
curl -s -X POST https://soc.bedrockcybersecurity.org/api/maintenance/active \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <key>" \
  -d '{
    "description": "Patching all PROD VMs — EventID 4624/7045 expected at volume",
    "start": "2026-05-25T02:00:00Z",
    "end": "2026-05-25T04:00:00Z"
  }' | jq .

Fields:

  • description — what work is being done; recorded in the audit trail.
  • start — ISO 8601 UTC timestamp. Can be now or in the future.
  • end — ISO 8601 UTC timestamp. No open-ended windows; end time is required.

Checking Active Windows

SecurityAgent calls GET /api/maintenance/active on every cycle. You can call it directly to confirm a window is active:

bash
curl -s https://soc.bedrockcybersecurity.org/api/maintenance/active \
  -H "X-API-Key: <key>" \
  | jq .

Response when a window is active:

json
{
  "active": true,
  "window": {
    "id": 12,
    "description": "Patching all PROD VMs",
    "start": "2026-05-25T02:00:00Z",
    "end": "2026-05-25T04:00:00Z"
  }
}

Response when no window is active:

json
{
  "active": false,
  "window": null
}

What Happens During a Maintenance Window

When GET /api/maintenance/active returns active: true, SecurityAgent:

  1. Logs a heartbeat (confirms agents ran normally).
  2. Skips incident creation for all findings from all environments.
  3. Does not send Telegram notifications.

Findings are still collected and stored — the suppression happens at the incident creation step, not at the finding ingestion step. You can query findings from a maintenance window period after the fact.

PostgreSQL: Maintenance Window History

sql
SELECT
  id,
  description,
  start_time,
  end_time,
  (end_time - start_time) AS duration,
  created_by
FROM maintenance_windows
ORDER BY start_time DESC
LIMIT 20;

Quick Reference

Which mechanism to use

ScenarioUse
PR merge to any infra repoAutomatic CM ticket (ingest-changes.yml)
Manual change outside PR flowManual CM ticket (POST /api/changes)
Ingest workflow failed after mergeManual CM ticket with pr_number set
Emergency production fixManual CM ticket first, then make the change
Multi-hour bulk maintenanceMaintenance window (POST /api/maintenance/active)
Recurring benign pattern (permanent)Known-good rule (requires Rory approval)

Key API Endpoints

POST /api/changes                — Create CM ticket (automated or manual)
GET  /api/changes/recent         — What SecurityAgent sees (last 2 hours)
GET  /api/changes?since=<ISO>    — Historical CM ticket query
GET  /api/changes/<id>           — Single ticket lookup
POST /api/maintenance/active     — Create maintenance window
GET  /api/maintenance/active     — Check if a window is currently active

Key Contacts

  • Final authority: Rory — all PRs, all architectural decisions, all emergency changes
  • SecOps platform: secops.bedrockcybersecurity.org (Azure AD SSO)
  • API key: Key Vault cirius-openai-kv-prod → secret secops-api-key
  • Workflow failures: GitHub Actions → ingest-changes.yml in the affected repo

Last updated: 2026-05-25
Owner: Rory
Related runbooks: secops-platform-guide.md, known-good-rule-management.md, incident-response.md
Related docs: cicd/pipeline-overview.md

Internal use only — Cirius Group