Docs

Appearance

Runbook: Cloud PC Rebuild (Kobe) ​

Purpose ​

This runbook describes how to rebuild the Kobe Cloud PC — the Ubuntu 24.04 EC2 instance in the AWS Prod account that runs the Kobe Claude Code agent. It is written so that an operator who has never touched this system can follow it and bring Kobe back online in under 2 hours from a cold start.

Use this runbook when Kobe is unrecoverable (cloud-init failure, package corruption, suspected compromise, or intentional greenfield rebuild). Diagnose-first is almost always the right call — see When to Rebuild vs. Diagnose below.

Overview ​

AttributeValue
PurposeRuns Kobe Claude Code agent — infrastructure automation, Telegram channel, SecOps integration
AMIUbuntu 24.04 LTS (Noble Numbat), latest Canonical
Instance typet3a.2xlarge
AWS accountProd — 807267566999
Regionus-west-2
VPC / subnetProd VPC, prod-workers subnet (10.50.2.0/24, us-west-2b)
Private IP10.50.2.8 — static, survives rebuilds
Public IPNone — no internet inbound
Access (primary)SSH via Twingate zero-trust overlay only
Access (break-glass)AWS Systems Manager Session Manager
IAM roleCloudPCRole (ReadOnlyAccess + SSM + state S3 read + kobe/* SM read)
Terraformaws-infra/prod/ec2_cloudpc.tf, aws-infra/prod/iam_cloudpc.tf
Bootstrapaws-infra/prod/scripts/kobe-bootstrap.sh (stored in S3, fetched by cloud-init)

Why the static private IP matters. 10.50.2.8 is pinned in Terraform (private_ip = "10.50.2.8"). Twingate resource definitions, SSH config entries, and internal DNS all point at this IP. A rebuild produces a new instance ID but the same IP — no Twingate reconfiguration is required.

What is and is not in Terraform. The EC2 instance, security group, IAM role, instance profile, and the minimal cloud-config (hostname + SSH authorized keys + bootstrap bootstrapper) are all Terraform-managed. The actual setup logic — packages, Claude Code, MCP servers, Cortex XDR, auditd, the kobe.service systemd unit — lives entirely in prod/scripts/kobe-bootstrap.sh. Bootstrap changes do NOT require a Terraform apply, only an S3 sync.

When to Rebuild vs. Diagnose ​

Rebuilding destroys local state (shell history, in-flight work directories under /home/ubuntu/, any uncommitted git changes across the seven cloned repos). Diagnose first unless one of the rebuild triggers applies.

SituationFirst action
Suspected compromise of hostSnapshot then rebuild — preserve disk for forensics, replace the host
cloud-init finished cleanly but a package is brokenDiagnose — sudo journalctl -u kobe.service, targeted fix
kobe.service flappingDiagnose — systemctl status kobe, read tmux session, check Telegram config
Disk full / unbootableRebuild — root is 50 GB ephemeral; no persistent data by design
cloud-init never completed on first bootRebuild (see Troubleshooting)
Cortex XDR agent desyncedDiagnose first — /opt/traps/bin/cytool reconnect force <key>
Claude Code OAuth refresh failed repeatedlyDiagnose — re-pull kobe/claude-credentials, restart kobe.service
You need a clean room for a testRebuild

If in doubt, open an SSM session (see SSM Break-Glass Access) and read /var/log/kobe-init.log and /var/log/cloud-init-output.log before making the call.

Preconditions ​

All of these must be true before you terminate the existing instance. Check each one.

1. Secrets Manager — all kobe/* secrets present in prod account (807267566999, us-west-2) ​

The bootstrap script reads these. If any are missing or expired, cloud-init will fail silently and Kobe will come up broken.

SecretPurpose
kobe/github-ssh-keyed25519 private key for Cirius-Group-Inc repos (root and ubuntu ~/.ssh/id_ed25519)
kobe/gh-tokenGitHub OAuth token for gh CLI (cirius-kobe-bot)
kobe/azure-sp-credentialsJSON client_id/client_secret/tenant_id for Azure PROD SP (tenant d477c9f8)
kobe/azure-dde-sp-credentialsJSON for DDE SP (tenant ff1c5d68)
kobe/anthropic-api-keyAnthropic API key → written to /home/ubuntu/.env_secrets
kobe/claude-credentialsContents of ~/.claude/.credentials.json (OAuth — refresh token auto-renews)
kobe/telegram-configTelegram plugin .env for Claude Code channels
kobe/telegram-bridgeTelegram bridge telegram.conf
kobe/pai-settingsPAI settings.json (v4.0.3+ — keep contextFiles: [])
kobe/aws-config~/.aws/config with all seven SSO profiles
kobe/secops-db-urlPostgres URL for the SecOps MCP server
kobe/ssh-key-azure-rsaAzure VM SSH key
kobe/ssh-key-ec2-keyec2-key.pem
kobe/ssh-key-id-rsaGeneric RSA private key
kobe/ssh-key-veeamVeeam backup target key

Quick check from any operator workstation with prod account credentials:

bash
aws secretsmanager list-secrets --region us-west-2 --profile prod --query "SecretList[?starts_with(Name, 'kobe/')].Name" --output table

Expect all 15 entries above. If any are missing, recreate them before rebuilding — cross-reference aws-infra/prod/ec2_cloudpc.tf header and prod/scripts/kobe-bootstrap.sh for their source.

2. Cross-account Secrets Manager access (main account 206820231356) ​

terraform/* and twingate/* secrets live in the main account. CloudPCRole in prod assumes a resource-based policy on each secret. This is one-time setup — should already be in place. If a new secret is added to either namespace, it needs a resource policy granting arn:aws:iam::807267566999:role/CloudPCRole the secretsmanager:GetSecretValue action.

3. S3 — bootstrap script synced to the install bucket ​

bash
aws s3 ls s3://cirius-archive-logs-prod/installers/kobe-bootstrap.sh --region us-west-2 --profile logging

If you changed aws-infra/prod/scripts/kobe-bootstrap.sh and have not synced it, the old version will run:

bash
aws s3 cp aws-infra/prod/scripts/kobe-bootstrap.sh \
  s3://cirius-archive-logs-prod/installers/kobe-bootstrap.sh \
  --region us-west-2 --profile logging

4. Ancillary installers in S3 ​

  • s3://cirius-archive-logs-prod/installers/hooks/hooks-backup.tar.gz — PAI hooks overlay
  • s3://cirius-archive-logs-prod/installers/cortex/cortex-install.tar.gz — Cortex XDR installer bundle (cortex-9.1.0.144052.deb + cortex.conf)
  • s3://config-bucket-206820231356/arctic-wolf/arcticwolfagent_*.deb — Arctic Wolf agent + customer.json

All should already exist. Verify with aws s3 ls before rebuilding.

5. Twingate resource still pointing at 10.50.2.8 ​

The static IP is intentional — Twingate resource config for Kobe should need no change. Confirm in the Twingate admin console that the kobe-cloud-pc resource target is 10.50.2.8 before continuing.

6. Operator has the right access ​

  • AWS SSO role in the prod account with read + SSM permissions (for verification and break-glass)
  • Write access to Cirius-Group-Inc/aws-infra (to open the PR that triggers apply)
  • Twingate access to the internal network (to test SSH post-rebuild)

Rebuild Procedure ​

Estimated wall-clock time: 90 minutes end to end (terraform apply: ~2 min, cloud-init + bootstrap: ~15–20 min, verification: ~15 min, contingency: balance).

Phase A — Prepare (5 min) ​

  1. Confirm all preconditions above.

  2. Pull latest aws-infra:

    bash
    cd ~/src/aws-infra && git checkout main && git pull --ff-only

  3. Review prod/ec2_cloudpc.tf and prod/scripts/kobe-bootstrap.sh to confirm no unexpected uncommitted diffs.

Phase B — Snapshot (optional, 5 min — only if diagnosing or preserving evidence) ​

If the rebuild is for diagnostics or suspected compromise, snapshot the root EBS volume before terminating. Skip this for routine rebuilds — the disk is stateless by design.

bash
INSTANCE_ID=$(terraform -chdir=prod output -raw cloudpc_instance_id)
VOL_ID=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --region us-west-2 --profile prod --query 'Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId' --output text)
aws ec2 create-snapshot --volume-id "$VOL_ID" --description "Kobe Cloud PC pre-rebuild $(date -u +%Y%m%dT%H%M%SZ)" --region us-west-2 --profile prod

Record the resulting SnapshotId. Snapshots are billed — delete once the investigation is complete.

Phase C — Trigger the rebuild (choose one path) ​

There are two supported paths. Both produce the same outcome: a new EC2 instance with a fresh root disk, the same static IP, and the same cloud-init.

Path 1 — Edit user_data and let Terraform replace (preferred for clean rebuild):

The instance is declared with user_data_replace_on_change = true. Any change to the user_data heredoc in ec2_cloudpc.tf (for example, bumping the DeployedOn tag, or touching a comment inside the heredoc) causes Terraform to destroy and recreate the instance on apply.

Minimal nudge:

hcl
tags = merge(module.tags.required_tags, {
  Name       = "Kobe-Cloud-PC"
  Owner      = "kobe"
  Lifecycle  = "permanent"
  DeployedOn = "YYYY-MM-DD<bump-suffix>"   # bump this
  PatchGroup = "cirius-ubuntu-patch"
})

The DeployedOn tag change alone will NOT trigger recreation — the tag is metadata. To force replacement, edit the cloud-config heredoc (for example, add a comment line # rebuild YYYY-MM-DD) so user_data_replace_on_change fires.

Open a PR against aws-infra, review the plan (expect "1 to destroy, 1 to add"), merge. CI applies via OIDC. Never terraform apply locally — the hard rule is branch → PR → CI → merge, always.

Path 2 — Terminate first, let the next apply recreate:

If you want to free the IP immediately (for example, because the old host is compromised and should be disconnected before the new one is ready), terminate via console or CLI:

bash
INSTANCE_ID=$(terraform -chdir=~/src/aws-infra/prod output -raw cloudpc_instance_id)
aws ec2 terminate-instances --instance-ids "$INSTANCE_ID" --region us-west-2 --profile prod

Then open a trivial PR on aws-infra (for example, a comment tweak in ec2_cloudpc.tf) so CI's terraform apply on merge recreates the instance. The next apply will detect the missing instance and rebuild it from the declared resource.

Hard rule reminder: Local Terraform is read-only (plan, show, state list). Applies happen only through GitHub Actions with OIDC after merge to main.

Phase D — Open the PR and merge (10 min) ​

  1. Branch from main in aws-infra:

    bash
    cd ~/src/aws-infra
git checkout main && git pull --ff-only
git checkout -b infra/cloudpc-rebuild-$(date -u +%Y%m%d)

  2. Make the minimal edit (comment in user_data heredoc, or whatever the rebuild requires).

  3. terraform fmt prod/ and run the standard PR pre-flight (terraform init / validate / plan).

  4. Commit, push, open a PR.

  5. Confirm CI posts a plan showing 1 to destroy, 1 to add for aws_instance.cloudpc (or 1 to add only, if Phase C Path 2 terminated the instance).

  6. Get Rory's review and merge. CI applies via OIDC.

Phase E — Wait for cloud-init + bootstrap (15–20 min) ​

The new instance comes up within 30 seconds. cloud-init runs immediately, downloads kobe-bootstrap.sh from S3 (with a 5-attempt retry loop to handle the brief IAM propagation race), and executes it. The full bootstrap typically takes 12–18 minutes — it installs Terraform, AWS CLI v2, gh CLI, Node.js 20, Claude Code, bun, auditd rules, PAI, MCP servers, every kobe/* secret materialization, all seven repo clones, Cortex XDR, Docker, git-secrets, Python libs, the Azure CLI venv, both Azure SP logins, and the Arctic Wolf agent.

While you wait, you can tail progress (see SSM section below for how to connect):

bash
sudo tail -f /var/log/kobe-init.log           # the 5 high-level echo lines from cloud-config
sudo tail -f /var/log/cloud-init-output.log   # everything kobe-bootstrap.sh does

Bootstrap is done when you see [kobe-bootstrap] HH:MM:SS bootstrap complete in the log.

Phase F — Verify (15 min) ​

See the Verification Checklist below. Run every check. Do not declare the rebuild complete until all pass.

SSM Break-Glass Access ​

When Twingate is down, or you need to see cloud-init while it is still running, use AWS Systems Manager Session Manager. CloudPCRole has AmazonSSMManagedInstanceCore attached; the SSM agent is present on the stock Ubuntu 24.04 AMI and registers within about 60 seconds of boot.

Get the instance ID:

bash
cd ~/src/aws-infra
terraform -chdir=prod output -raw cloudpc_instance_id
# or, from the outputs module:
terraform -chdir=prod output cloudpc_ssm_connect

Connect:

bash
aws ssm start-session --target i-0xxxxxxxxxxxxxxxx --region us-west-2 --profile prod

You land as ssm-user. Elevate with sudo -iu ubuntu (or sudo -i for root) to read bootstrap logs and run verification commands.

If SSM does not list the instance:

bash
aws ssm describe-instance-information --region us-west-2 --profile prod \
  --filters "Key=InstanceIds,Values=i-0xxxxxxxxxxxxxxxx"

An empty result means the SSM agent has not registered yet. Wait 60 seconds and retry. If it never appears, the instance lost internet egress or IMDSv2 is failing — check the VPC route and the instance profile attachment.

Verification Checklist ​

Run all of these before declaring rebuild complete. Most can be executed in sequence from a Twingate-connected SSH session; a couple are from the operator workstation.

Connect first:

bash
ssh ubuntu@10.50.2.8    # via Twingate

If SSH fails, fall back to SSM (see above) and diagnose.

1. Instance is healthy ​

bash
uptime
systemctl is-system-running   # expect "running" (or "degraded" only for the known Cortex unit)

2. cloud-init completed successfully ​

bash
sudo tail -5 /var/log/kobe-init.log
# Expect:  "[kobe-init] <ts> bootstrap complete"
sudo grep -E '^\[kobe-bootstrap\].*bootstrap complete' /var/log/cloud-init-output.log

3. kobe.service is running ​

bash
systemctl status kobe --no-pager
# Expect: active (running); tmux session "kobe" attached
tmux ls

4. Claude Code installed and authenticated ​

bash
claude --version
# Credentials file present
test -s /home/ubuntu/.claude/.credentials.json && echo "creds ok"

5. check-auth green across AWS / Azure PROD / Azure DDE / GitHub ​

bash
source ~/.bashrc_custom
check-auth

All four sections should return identity info. Any "NOT LOGGED IN" or gh auth status failure = fix before closing out.

6. Twingate — SSH works from the operator workstation ​

From your laptop (connected to Twingate):

bash
ssh ubuntu@10.50.2.8 "hostname && date -u"
# Expect: Kobe-Cloud-PC <current UTC>

7. Telegram channel bridge — message round-trip ​

From the authorized operator Telegram account, send ping to the Kobe bot chat. Expect a reply within ~10 seconds. On the host:

bash
sudo journalctl -u kobe --since "2 minutes ago" --no-pager | grep -i telegram

If no reply, check that /home/ubuntu/.claude/channels/telegram/.env exists and is non-empty, then sudo systemctl restart kobe.

8. Cortex XDR agent enrolled ​

bash
sudo /opt/traps/bin/cytool status
# Expect: Connection status Connected, Policy status Applied

In the Cortex console, confirm Kobe-Cloud-PC appears in the Endpoints list with a recent check-in. The old instance entry can be deleted manually from the console; Cortex does not auto-dedupe on hostname.

9. Arctic Wolf agent running ​

bash
systemctl status arcticwolfagent --no-pager | head -5

10. auditd active (kill chain Phase 1 coverage) ​

bash
sudo systemctl is-active auditd      # expect "active"
sudo auditctl -l | grep process_execution
# Expect: rules for execve on arch=b64 and arch=b32

11. Repos cloned and git identity set ​

bash
ls /home/ubuntu/src/
# Expect: aws-infra azure-infra azure-dde-infra ops-automation palo-alto-configs bedrock-docs bedrock-hub
git -C /home/ubuntu/src/aws-infra config user.email
# Expect: kobe-bot@ciriusgroup.com

12. SecOps API reachable — Kobe can pick up stories ​

bash
curl -sS https://soc.bedrockcybersecurity.org/api/stories?status=BACKLOG -H "X-API-Key: $(aws secretsmanager get-secret-value --secret-id kobe/secops-api-key --region us-west-2 --query SecretString --output text)" | jq 'length'

Any non-error response proves network egress + auth are healthy.

13. SSM break-glass still works from the operator side ​

Disconnect Twingate briefly and confirm aws ssm start-session --target i-... --region us-west-2 still lands you on the new host. Prevents discovering a broken SSM path only during the next real incident.

Estimated Time ​

PhaseActivityTime
APrepare — confirm preconditions, pull repos5 min
BOptional snapshot (diagnostic rebuilds only)5 min
CEdit user_data or terminate5 min
DOpen PR, CI plan, review, merge, CI apply15 min
EWait for cloud-init + kobe-bootstrap.sh20 min
FRun the full Verification Checklist15 min
—Contingency / troubleshooting buffer30 min
Total≤ 95 min (≤ 2 h target met)

Troubleshooting ​

cloud-init appears stuck (no progress in /var/log/kobe-init.log after 5 min) ​

Open SSM (Twingate will not work yet — no bootstrap = no user configured):

bash
sudo cloud-init status --long
sudo tail -100 /var/log/cloud-init.log

Most common causes:

  • S3 download failed — the retry loop in cloud-config gives 5 tries; after that it gives up. Confirm s3://cirius-archive-logs-prod/installers/kobe-bootstrap.sh exists and CloudPCRole has ReadOnlyAccess.
  • IMDSv2 not getting tokens — symptom is AWS CLI v1 (installed by apt in pre-bootstrap) failing credential discovery. The cloud-config installs v2 explicitly to avoid this. If the v2 install itself failed (for example, curl egress blocked), the whole chain stops.
  • VPC egress blocked — the NVA route may be dropping bootstrap traffic. Check flow logs in cirius-logging-law-central.

Bootstrap finishes but kobe.service fails to start ​

bash
sudo journalctl -u kobe -n 200 --no-pager

Usual suspects: expired kobe/claude-credentials refresh token, bad JSON in kobe/telegram-config, or the tmux session already existing (systemd Type=forking with stale session). Fix with:

bash
sudo -u ubuntu tmux kill-session -t kobe 2>/dev/null || true
sudo systemctl restart kobe

Cortex XDR shows the old host, not the new one ​

Cortex identifies endpoints by a combination of hostname and internal UUID. A fresh host with the same hostname (Kobe-Cloud-PC) will register as a new endpoint. The old entry in the Cortex console stays visible as "Disconnected" — delete it manually once you have confirmed the new endpoint is Connected + Applied.

If the new host does not register after 5 minutes:

bash
sudo /opt/traps/bin/cytool reconnect force f64d4b7da6774ccca9adb4db22112652

The key is the Cirius Cortex tenant ID, also hard-coded in the bootstrap script.

GitHub SSH auth fails after rebuild ​

bash
sudo -u ubuntu ssh -T git@github.com
# Expect: "Hi cirius-kobe-bot! You've successfully authenticated..."

If not, kobe/github-ssh-key is likely rotated or wrong. Re-put the correct ed25519 private key into Secrets Manager and rerun:

bash
sudo bash /tmp/bootstrap.sh   # idempotent; will overwrite .ssh keys from SM

Twingate resource not reachable (10.50.2.8 timeouts) ​

  • Confirm new instance is running in the AWS console
  • Confirm security group cloudpc still allows SSH from the firewall VPC CIDR (var.firewall_vpc_cidr)
  • Confirm Twingate connectors (mutant-nuthatch, papaya-manul) are Connected in the Twingate admin
  • Flush Twingate client resources on your laptop (sign out / sign back in)

Rollback ​

There is no rollback — a rebuild is itself the recovery path. If the new instance is worse than the old one and you snapshotted in Phase B, the recovery is: restore a fresh EBS volume from the snapshot, detach the new root, attach the restored volume as root, and reboot. In practice this is almost never the right call — a second rebuild from a clean kobe-bootstrap.sh is faster and more predictable.

Document History ​

DateChangeAuthor
April 2026Initial runbook — CLOUDPC-013; documents terminate / apply / verify flow, SSM break-glass, full verification checklist, <2 h targetKobe

Internal use only — Cirius Group