Runbook Factory

Story: [ORCH-082]
Owner: Dreaming automation
Frequency: Monthly (1st day of month, 08:00 UTC)
Workflow: .github/workflows/runbook-factory.yml
Script: scripts/runbook-factory.ts

Purpose

bedrock-docs is the runbook factory. Primary job: keep documentation current with what the platform actually does. Secondary job: generate PDFs via ProductDocs.

Never write documentation that contradicts what the code actually does.
Always pull from source of truth. Never guess.

Sources of Truth

The runbook factory pulls from 7 sources:

Source	What It Checks	Drift Type
Langfuse incident history	90-day incident patterns	`missing_incident_pattern` — recurring incidents (3+) without runbook
GitHub merged PRs	Infrastructure changes across all repos	`outdated_procedure` — new resources/services not reflected in docs
OpenAPI specs	SOC/GRC API references	Handled by `sync-api-refs.yml` — factory verifies reachability
Health signal log	30-day degraded/down signals	`outdated_procedure` — recurring issues (5+) needing troubleshooting section
Orchestrator job history	30-day job failures	`outdated_procedure` — job failures indicating runbook gaps
AWS Secrets Manager inventory	All 7 AWS accounts	`missing_credential` — undocumented secrets in `security-footprint.md`
Security group rules	All AWS accounts with egress rules	`missing_network_rule` — undocumented SGs in `security-footprint.md`

Drift Findings

Each finding includes:

typescript

{
  type: "missing_credential" | "missing_access_path" | "missing_network_rule" | "outdated_procedure" | "missing_incident_pattern",
  severity: "HIGH" | "MEDIUM" | "LOW",
  description: "what's missing or outdated",
  source_of_truth: "where the canonical data lives",
  affected_file: "which runbook needs updating (if known)"
}

Severity Levels

HIGH — Undocumented credentials, access paths, or network rules (security footprint drift)
MEDIUM — Recurring incidents/issues or orchestrator failures without runbook coverage
LOW — Recent infrastructure changes that may need doc updates

Ownership Boundary: Dreaming vs SOC

Credential expiration monitoring is owned by the SOC maintenance agent, NOT by Dreaming.

SOC maintenance agent: Monitors credential expiry, fires alerts, acts on expirations
Dreaming (runbook factory): Audits security-footprint.md for accuracy and completeness, verifies SOC alerts resulted in completed rotation stories

No duplication of SOC monitoring here.

Workflow Behavior

Monthly trigger: 1st day of month at 08:00 UTC
Manual trigger: workflow_dispatch for on-demand audits
Checks all 7 sources (fail-open if any source unreachable)
Generates report: runbook-factory-report.json with all findings
Opens PR if drift detected:
- Branch: auto/runbook-factory-<run_id>
- Title: [ORCH-082] Runbook Factory monthly audit — N findings
- Body: Drift summary with severity breakdown
- Includes report JSON for review

Manual Execution

Run locally for testing:

bash

bun run runbook-factory
# or
bun scripts/runbook-factory.ts

Prerequisites:

AWS CLI configured with read access to Secrets Manager and EC2
gh CLI authenticated
Environment variables (optional):
- SOC_ORIGIN — SOC platform URL (defaults to ca-soc-prod)
- GRC_ORIGIN — GRC platform URL (defaults to ca-grc-prod)

Report Structure

json

{
  "timestamp": "2026-07-01T08:00:00.000Z",
  "sources_checked": {
    "langfuse": true,
    "github": true,
    "openapi": true,
    "health_signals": true,
    "orchestrator_history": true,
    "secrets_manager": true,
    "security_groups": true
  },
  "drift_findings": [
    {
      "type": "missing_credential",
      "severity": "HIGH",
      "description": "Undocumented secret in Logging account: /bedrock/api-key",
      "source_of_truth": "AWS Secrets Manager (038901680748)",
      "affected_file": "architecture/security-footprint.md"
    }
  ],
  "runbooks_regenerated": [],
  "errors": []
}

Responding to Drift

When a runbook factory PR is opened:

Review runbook-factory-report.json for all findings
For HIGH severity findings (undocumented credentials/network rules):
- Update architecture/security-footprint.md immediately
- Verify with Rory if new access path is authorized
For MEDIUM severity findings (recurring incidents/issues):
- Check if pattern is already known and acceptable
- Add troubleshooting section to relevant runbook if needed
- Create incident response procedure if pattern is critical
For LOW severity findings (recent infra changes):
- Review merged PRs for context
- Update affected runbooks if changes impact documented procedures
- Close as "no action needed" if change is internal and doesn't affect docs

Integration with Other Workflows

Workflow	Relationship
`sync-api-refs.yml`	Handles OpenAPI spec regeneration weekly — runbook factory verifies reachability only
`claude-synthesis.yml`	Regenerates CLAUDE.md synthesis on pushes — runbook factory focuses on runbooks
`freshness-check.yml`	Flags stale pages (180+ days) — runbook factory flags drift from platform reality
`baseline-scan.yml`	Code embedding baseline — runbook factory audits documentation content

Example Findings

Missing Credential

json

{
  "type": "missing_credential",
  "severity": "HIGH",
  "description": "Undocumented secret in bedrock-platform account: /bedrock/telegram-bot-token",
  "source_of_truth": "AWS Secrets Manager (863609217450)",
  "affected_file": "architecture/security-footprint.md"
}

Action: Add credential to security footprint with purpose, rotation schedule, and owner.

Recurring Incident Pattern

json

{
  "type": "missing_incident_pattern",
  "severity": "MEDIUM",
  "description": "Recurring incident pattern detected: PROD_authentication_mfa_bypass (5 occurrences in 90 days) — no runbook found",
  "source_of_truth": "Langfuse incident history"
}

Action: Create incident response procedure for MFA bypass attempts, add to incident-response.md.

Outdated Procedure

json

{
  "type": "outdated_procedure",
  "severity": "LOW",
  "description": "Recent infrastructure PR may require doc update: Add Cortex XDR agent to Cloud PC instances",
  "source_of_truth": "https://github.com/Bedrock-Cybersecurity/aws-infra/pull/123"
}

Action: Review PR, update cloud-pc-operations.md if new procedure affects documented workflows.

Troubleshooting

Source Unreachable

If a source is unreachable (e.g., SOC platform down, AWS credentials invalid), the workflow continues with other checks (fail-open design). Errors are logged in report.errors[].

Check:

SOC/GRC platform health
AWS OIDC role permissions
GitHub token scopes

False Positives

If the factory flags a credential/rule that IS documented:

Verify the documentation uses the exact resource ID/name from AWS
Check if the credential is in a different section (e.g., Azure vs AWS)
Update the factory script search patterns if needed

High Severity Findings on First Run

The first run may flag many HIGH severity findings if security-footprint.md is incomplete. This is expected — work through findings in priority order:

Break-glass accounts and root credentials
Service-to-service API keys
Database connection strings
External integration tokens

Metrics

Track these over time to measure documentation health:

Drift findings per month (trend should be downward after first few runs)
HIGH severity findings (should stay at 0 after initial cleanup)
Time to resolve findings (PR open → merged)
Sources successfully checked (7/7 is healthy)

Future Enhancements

Potential additions (not yet implemented):

PDF generation tracking: Verify ProductDocs PDFs are regenerated when runbooks change
Cross-reference validation: Ensure runbook links point to valid pages
Procedure verification: Test documented procedures against live platform (e.g., verify backup restore steps actually work)
Compliance mapping: Flag when HIPAA/HITRUST controls reference outdated procedures

Security Monitoring Architecture — credential and network access inventory
Incident Response — incident handling procedures
Common Tasks — operational runbooks
CLAUDE.md (repo root) — automation patterns and lessons learned

Last updated: 2026-06-30
Status: Active (ORCH-082 in progress)

Runbook Factory ​

Purpose ​

Sources of Truth ​

Drift Findings ​

Severity Levels ​

Ownership Boundary: Dreaming vs SOC ​

Workflow Behavior ​

Manual Execution ​

Report Structure ​

Responding to Drift ​

Integration with Other Workflows ​

Example Findings ​

Missing Credential ​

Recurring Incident Pattern ​

Outdated Procedure ​

Troubleshooting ​

Source Unreachable ​

False Positives ​

High Severity Findings on First Run ​

Metrics ​

Future Enhancements ​

Related Documentation ​

Runbook Factory

Purpose

Sources of Truth

Drift Findings

Severity Levels

Ownership Boundary: Dreaming vs SOC

Workflow Behavior

Manual Execution

Report Structure

Responding to Drift

Integration with Other Workflows

Example Findings

Missing Credential

Recurring Incident Pattern

Outdated Procedure

Troubleshooting

Source Unreachable

False Positives

High Severity Findings on First Run

Metrics

Future Enhancements

Related Documentation