Key Vault Secrets Workflow

Scope: cirius-openai-kv-prod, Azure PROD tenant d477c9f8 (ciriusgroup.com) Platform: FastAPI on Container Apps (ca-soc-prod, rg-logging-logs, logging subscription) Auth model: Managed identity at runtime; OIDC federation in GitHub Actions; no stored credentials anywhere

1. Creating a New Secret

Portal

Navigate to Key Vault → cirius-openai-kv-prod
Secrets → Generate/Import
Upload method: Manual
Name: use kebab-case, descriptive — e.g. openai-api-key, postgres-password-prod
Value: paste the secret value
Set activation/expiration dates if appropriate (recommend expiration for API keys)
Create

CLI

bash

# Authenticate first
az login --tenant d477c9f8-xxxx-xxxx-xxxx-xxxxxxxxxxxx

# Set subscription context (logging subscription hosts the KV)
az account set --subscription "<logging-subscription-id>"

# Create a new secret
az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "sk-prod-xxxxxxxxxxxx"

# Create with expiration (recommended for API keys)
az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "sk-prod-xxxxxxxxxxxx" \
  --expires "2027-01-01T00:00:00Z"

# Verify it was created
az keyvault secret show \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --query "{id:id, enabled:attributes.enabled, expires:attributes.expires}" \
  --output table

Naming conventions:

<service>-<purpose>-<env> — e.g. openai-api-key-prod, postgres-password-prod
Never include the actual secret value in the name
Avoid generic names like api-key or password — they become ambiguous over time

2. Granting Access

Managed Identity (preferred — runtime access from Container Apps)

The FastAPI app running on ca-soc-prod uses a system-assigned or user-assigned managed identity. No credential management required.

Check what identity the Container App has:

bash

az containerapp show \
  --name ca-soc-prod \
  --resource-group rg-logging-logs \
  --query "identity" \
  --output json

If system-assigned identity is not yet enabled:

bash

az containerapp identity assign \
  --name ca-soc-prod \
  --resource-group rg-logging-logs \
  --system-assigned

Capture the identity's principal ID:

bash

PRINCIPAL_ID=$(az containerapp show \
  --name ca-soc-prod \
  --resource-group rg-logging-logs \
  --query "identity.principalId" \
  --output tsv)

echo "Principal ID: $PRINCIPAL_ID"

Grant Key Vault Secrets User (read-only — correct for runtime):

bash

KV_ID=$(az keyvault show \
  --name cirius-openai-kv-prod \
  --query id \
  --output tsv)

az role assignment create \
  --assignee "$PRINCIPAL_ID" \
  --role "Key Vault Secrets User" \
  --scope "$KV_ID"

Service Principal (for automation, CI/CD, or cross-tenant access)

Only use a service principal when managed identity is not available (e.g., an external tool, a non-Azure service).

bash

# Create SP
az ad sp create-for-rbac \
  --name sp-keyvault-readonly \
  --skip-assignment

# Capture the appId from the output, then assign role
SP_APP_ID="<appId-from-output>"

az role assignment create \
  --assignee "$SP_APP_ID" \
  --role "Key Vault Secrets User" \
  --scope "$KV_ID"

Store the SP credentials in Key Vault itself (or AWS Secrets Manager) — never in GitHub secrets, .env files, or code.

RBAC Roles Reference

Role	Can Do	Use For
Key Vault Secrets User	Read secret values	App runtime, read-only automation
Key Vault Secrets Officer	CRUD secrets, no admin	Engineers managing secrets
Key Vault Administrator	Full control including RBAC	Break-glass, KV provisioning only
Key Vault Reader	Read metadata only, not values	Audit tools, Terraform plan

Principle of least privilege: Container Apps get Key Vault Secrets User. Engineers doing secret management get Key Vault Secrets Officer. No one gets Key Vault Administrator for day-to-day work.

3. Referencing a Secret in Terraform

Use the azurerm_key_vault_secret data source to read secrets during plan/apply. The Terraform runner authenticates via OIDC (see section 5) and needs Key Vault Secrets User on the vault.

hcl

# Fetch the KV resource
data "azurerm_key_vault" "prod" {
  name                = "cirius-openai-kv-prod"
  resource_group_name = "rg-logging-logs"
}

# Read a specific secret
data "azurerm_key_vault_secret" "openai_api_key" {
  name         = "openai-api-key"
  key_vault_id = data.azurerm_key_vault.prod.id
}

# Use the secret value in a resource
resource "azurerm_container_app" "soc" {
  # ... other config ...

  template {
    container {
      name  = "soc-api"
      image = "ciriusgroupacr.azurecr.io/soc-api:latest"

      env {
        name  = "OPENAI_API_KEY"
        # Reference the value directly — Terraform reads it at apply time
        value = data.azurerm_key_vault_secret.openai_api_key.value
      }
    }
  }
}

Important: data.azurerm_key_vault_secret.*.value is marked sensitive in the Terraform state. The state file itself contains the plaintext value — ensure state is stored in a backend (Azure Storage with encryption) and access to state is restricted.

Preferred approach for Container Apps — use secretsRef instead (see section 4). This avoids the secret value ever touching Terraform state.

4. Referencing a Secret in Container Apps

Container Apps has native Key Vault integration. The app reads the secret directly from Key Vault at startup via its managed identity. The secret value never passes through Terraform state.

Method A: secretsRef (Recommended)

The Container App pulls the secret from Key Vault and injects it as an environment variable. The managed identity on the app must have Key Vault Secrets User.

Terraform config:

hcl

resource "azurerm_container_app" "soc" {
  name                         = "ca-soc-prod"
  resource_group_name          = "rg-logging-logs"
  container_app_environment_id = azurerm_container_app_environment.prod.id
  revision_mode                = "Single"

  identity {
    type = "SystemAssigned"
  }

  secret {
    name                = "openai-api-key"
    key_vault_secret_id = data.azurerm_key_vault_secret.openai_api_key.versionless_id
    identity            = "System"
  }

  template {
    container {
      name   = "soc-api"
      image  = "ciriusgroupacr.azurecr.io/soc-api:latest"
      cpu    = 0.5
      memory = "1Gi"

      env {
        name        = "OPENAI_API_KEY"
        secret_name = "openai-api-key"  # references the secret block above
      }
    }
  }
}

Note: versionless_id omits the version GUID — the app always resolves the current version. Use this unless you need to pin to a specific version.

Verify the secret reference via CLI:

bash

az containerapp secret list \
  --name ca-soc-prod \
  --resource-group rg-logging-logs \
  --output table

Method B: Direct SDK call in Python (runtime fetch)

For secrets that change frequently or are fetched conditionally:

python

from azure.identity import ManagedIdentityCredential
from azure.keyvault.secrets import SecretClient

credential = ManagedIdentityCredential()
client = SecretClient(
    vault_url="https://cirius-openai-kv-prod.vault.azure.net/",
    credential=credential
)

secret = client.get_secret("openai-api-key")
api_key = secret.value

This works because the Container App's managed identity has Key Vault Secrets User. No client ID, tenant ID, or credential is needed in the code.

Triggering a new revision after secret update

When a secret value is updated in Key Vault, Container Apps using secretsRef do not automatically pick up the new value. You must force a new revision:

bash

# Update the secret reference to force revision creation
az containerapp update \
  --name ca-soc-prod \
  --resource-group rg-logging-logs \
  --set-env-vars "FORCE_REDEPLOY=$(date +%s)"

# Or create a new revision directly
az containerapp revision copy \
  --name ca-soc-prod \
  --resource-group rg-logging-logs

5. GitHub Actions — Why Secrets Are Not Stored There

GitHub Actions authenticates to Azure using OIDC federation. There are no Azure credentials (client secret, certificate, or SAS token) stored in GitHub.

How it works:

GitHub generates a short-lived OIDC token at workflow runtime
The token is exchanged with Azure AD for an access token scoped to the federated credential
The workflow runs with the permissions of the service principal, for the duration of that run only

Workflow setup:

yaml

permissions:
  id-token: write   # required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Azure Login (OIDC)
        uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}          # not a secret — public SP app ID
          tenant-id: ${{ vars.AZURE_TENANT_ID }}          # not a secret
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}  # not a secret

      - name: Terraform Apply
        run: terraform apply -auto-approve

vars.* are GitHub Actions variables (not secrets) — they are not masked and not sensitive. The actual Azure credential is never stored anywhere; it is minted fresh per run.

What this means for Key Vault: GitHub Actions workflows do not need Key Vault access for most operations. The Terraform runner reads secrets during apply when needed (via the SP's RBAC assignment), but the secret values are not stored in GitHub.

If a workflow genuinely needs a secret at runtime (rare), use the Azure CLI to fetch it within the job after OIDC login — do not store it in a GitHub secret:

yaml

- name: Fetch secret for use in this job
  run: |
    SECRET_VALUE=$(az keyvault secret show \
      --vault-name cirius-openai-kv-prod \
      --name my-secret \
      --query value \
      --output tsv)
    echo "::add-mask::$SECRET_VALUE"
    echo "MY_SECRET=$SECRET_VALUE" >> $GITHUB_ENV

6. Secret Rotation Procedure

Standard rotation — planned update with zero downtime.

Step 1: Generate the new secret value

Generate the new credential from the source system (OpenAI dashboard, database admin console, etc.). Do not yet revoke the old one.

Step 2: Create a new version in Key Vault

bash

# Setting a new value creates a new version; the old version is preserved
az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "sk-prod-new-xxxxxxxxxxxx"

# Confirm the new version is current
az keyvault secret show \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --query "{version:id, enabled:attributes.enabled}" \
  --output json

Step 3: Force apps to pick up the new version

bash

# Restart Container App to load new secret (secretsRef method)
az containerapp revision copy \
  --name ca-soc-prod \
  --resource-group rg-logging-logs

# Confirm the new revision is active
az containerapp revision list \
  --name ca-soc-prod \
  --resource-group rg-logging-logs \
  --query "[].{name:name, active:properties.active, created:properties.createdTime}" \
  --output table

Step 4: Verify the app is working with the new secret

bash

# Check Container App logs for errors after restart
az containerapp logs show \
  --name ca-soc-prod \
  --resource-group rg-logging-logs \
  --tail 50

# If the app has a health endpoint
curl -s https://ca-soc-prod.<env>.azurecontainerapps.io/health

Watch for authentication errors (401/403 from the downstream service) which would indicate the new secret was not accepted.

Step 5: Disable the old version

Once you confirm the app is working with the new version, disable the old version in Key Vault. Do not delete it yet — disabling preserves audit history and allows rollback.

bash

# List versions to find the old version ID
az keyvault secret list-versions \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --output table

# Disable the old version (replace <old-version-id> with the version GUID)
az keyvault secret set-attributes \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --version <old-version-id> \
  --enabled false

Step 6: Revoke the old credential at the source

Return to the source system (OpenAI, database, etc.) and revoke or delete the old credential. This is the final step — the old value is now dead.

7. Emergency Rotation

Use when a secret is suspected or confirmed compromised (leaked in logs, exposed in code, phishing, insider threat).

Speed is the priority. Do not wait for change windows.

Immediate steps (target: under 15 minutes)

bash

# Step 1: Revoke the old credential at the SOURCE SYSTEM FIRST
# This kills the live credential before an attacker can use it further.
# Go to OpenAI/database/whatever issued it and delete/revoke NOW.
# Do this before anything else.

# Step 2: Create the new secret version in Key Vault
az keyvault secret set \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --value "sk-prod-emergency-new-xxxxxxxxxxxx"

# Step 3: Force Container App restart to pick up new value
az containerapp revision copy \
  --name ca-soc-prod \
  --resource-group rg-logging-logs

# Step 4: Disable ALL previous versions of the secret
# List all versions first
az keyvault secret list-versions \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --output tsv --query "[].{id:id, enabled:attributes.enabled}"

# Disable each old version
az keyvault secret set-attributes \
  --vault-name cirius-openai-kv-prod \
  --name openai-api-key \
  --version <version-id> \
  --enabled false

Scope the blast radius

bash

# Find all role assignments on this Key Vault — who has access?
az role assignment list \
  --scope "$(az keyvault show --name cirius-openai-kv-prod --query id --output tsv)" \
  --output table

# Check Key Vault access logs in Log Analytics (see section 8)
# Look for unexpected identities or IPs accessing the compromised secret

Follow-up (within 24 hours)

File an internal incident ticket
Document timeline: when exposure likely began, what was rotated, what the secret accessed
Review KQL audit logs (section 8) for the period from suspected exposure to rotation
Confirm no other secrets in the same vault were accessed by the same principal
If the breach originated from code (secret committed to git): remove from git history, rotate all secrets that were in that repo, audit who had read access to the repo

8. Audit — KQL Query for Key Vault Access

Log Analytics workspace: cirius-logging-law-central (logging subscription, rg-logging-logs)

Key Vault diagnostic logs must be forwarded to this workspace. Verify this is configured:

bash

az monitor diagnostic-settings list \
  --resource "$(az keyvault show --name cirius-openai-kv-prod --query id --output tsv)" \
  --output table

If not configured:

bash

LAW_ID=$(az monitor log-analytics workspace show \
  --workspace-name cirius-logging-law-central \
  --resource-group rg-logging-logs \
  --query id \
  --output tsv)

az monitor diagnostic-settings create \
  --name kv-to-law \
  --resource "$(az keyvault show --name cirius-openai-kv-prod --query id --output tsv)" \
  --workspace "$LAW_ID" \
  --logs '[{"category": "AuditEvent", "enabled": true}]'

KQL — All secret operations in the last 7 days

kql

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where ResourceType == "VAULTS"
| where Resource == "CIRIUS-OPENAI-KV-PROD"
| where TimeGenerated > ago(7d)
| project
    TimeGenerated,
    OperationName,
    ResultType,
    CallerIPAddress,
    identity_claim_oid_g,
    identity_claim_upn_s,
    requestUri_s
| sort by TimeGenerated desc

KQL — Access to a specific secret

kql

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where Resource == "CIRIUS-OPENAI-KV-PROD"
| where requestUri_s contains "openai-api-key"
| where TimeGenerated > ago(30d)
| project
    TimeGenerated,
    OperationName,
    ResultType,
    CallerIPAddress,
    identity_claim_upn_s,
    identity_claim_oid_g,
    requestUri_s
| sort by TimeGenerated desc

KQL — Failed access attempts (unauthorized or denied)

kql

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where Resource == "CIRIUS-OPENAI-KV-PROD"
| where ResultType != "Success"
| where TimeGenerated > ago(7d)
| project
    TimeGenerated,
    OperationName,
    ResultType,
    ResultDescription,
    CallerIPAddress,
    identity_claim_upn_s,
    requestUri_s
| sort by TimeGenerated desc

KQL — Access by a specific identity (use OID for managed identities)

kql

// Replace the OID with the managed identity's principal ID
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where Resource == "CIRIUS-OPENAI-KV-PROD"
| where identity_claim_oid_g == "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
| where TimeGenerated > ago(30d)
| project TimeGenerated, OperationName, ResultType, requestUri_s
| sort by TimeGenerated desc

KQL — Secret access volume by caller (detect anomalous access patterns)

kql

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where Resource == "CIRIUS-OPENAI-KV-PROD"
| where OperationName == "SecretGet"
| where TimeGenerated > ago(24h)
| summarize
    AccessCount = count(),
    Secrets = make_set(tostring(split(requestUri_s, "/secrets/")[1]))
    by identity_claim_upn_s, identity_claim_oid_g, CallerIPAddress
| sort by AccessCount desc

9. What Lives in Key Vault vs. What Is a Managed Identity

Decision guide — one question first: Can the service authenticate itself to Azure AD without a credential?

If yes → managed identity. No secret needed. If no → Key Vault.

Use Managed Identity For

Scenario	Why
Container App reading from Azure Blob Storage	App is Azure-native; identity is the credential
Container App reading from Key Vault itself	Managed identity fetches secrets without a secret
Container App calling Azure OpenAI (if using Azure-hosted endpoint)	Azure RBAC handles authorization
Terraform runner (GitHub OIDC) accessing Azure resources	OIDC token is the credential; nothing stored
Container App writing to Azure Service Bus	RBAC role on the Service Bus namespace
Container App reading from Azure SQL (with Entra auth enabled)	Entra token; no password needed

Managed identity is the right answer for any Azure-to-Azure communication where the target service supports Entra-based authentication.

Use Key Vault For

Secret	Why
OpenAI API key (api.openai.com — external)	External service; does not support managed identity
Stripe API key	External service
Sendgrid / SMTP credentials	External service
Third-party webhook signing secrets	External service
PostgreSQL password (if not using Entra auth)	Traditional credential; no identity federation
Any third-party SaaS credential	External; no Azure identity integration
Encryption keys used in application logic	Not an identity concern; Key Vault Keys is appropriate

What Never Goes Anywhere Except Key Vault

Do not put secrets in environment variables set at build time (baked into image)
Do not put secrets in Terraform .tfvars files committed to git
Do not put secrets in GitHub Actions secrets (use OIDC + runtime KV fetch if needed)
Do not put secrets in application config files (config.yaml, .env) committed or deployed without KV reference
Do not log secret values — log secret names only

Quick Reference Decision Tree

Is the caller an Azure resource (Container App, VM, Function)?
  └── Yes → Does the target service support Azure AD / Entra auth?
              └── Yes → Use managed identity. Done. No secret.
              └── No  → Store credential in Key Vault. Fetch via managed identity at runtime.
  └── No  → Is the caller GitHub Actions?
              └── Yes → Use OIDC. No stored secret.
              └── No  → Store SP credential in Key Vault. Access via SP that reads KV.

Quick Reference

Task	Command
List all secrets	`az keyvault secret list --vault-name cirius-openai-kv-prod --output table`
Show secret value	`az keyvault secret show --vault-name cirius-openai-kv-prod --name <name> --query value --output tsv`
List secret versions	`az keyvault secret list-versions --vault-name cirius-openai-kv-prod --name <name> --output table`
Disable a version	`az keyvault secret set-attributes --vault-name cirius-openai-kv-prod --name <name> --version <id> --enabled false`
List role assignments on KV	`az role assignment list --scope <kv-resource-id> --output table`
Check Container App secrets	`az containerapp secret list --name ca-soc-prod --resource-group rg-logging-logs`
Force new revision	`az containerapp revision copy --name ca-soc-prod --resource-group rg-logging-logs`
Tail Container App logs	`az containerapp logs show --name ca-soc-prod --resource-group rg-logging-logs --tail 100`

Key Vault Secrets Workflow ​

1. Creating a New Secret ​

Portal ​

CLI ​

2. Granting Access ​

Managed Identity (preferred — runtime access from Container Apps) ​

Service Principal (for automation, CI/CD, or cross-tenant access) ​

RBAC Roles Reference ​

3. Referencing a Secret in Terraform ​

4. Referencing a Secret in Container Apps ​

Method A: secretsRef (Recommended) ​

Method B: Direct SDK call in Python (runtime fetch) ​

Triggering a new revision after secret update ​

5. GitHub Actions — Why Secrets Are Not Stored There ​

6. Secret Rotation Procedure ​

Step 1: Generate the new secret value ​

Step 2: Create a new version in Key Vault ​

Step 3: Force apps to pick up the new version ​

Step 4: Verify the app is working with the new secret ​

Step 5: Disable the old version ​

Step 6: Revoke the old credential at the source ​

7. Emergency Rotation ​

Immediate steps (target: under 15 minutes) ​

Scope the blast radius ​

Follow-up (within 24 hours) ​

8. Audit — KQL Query for Key Vault Access ​

KQL — All secret operations in the last 7 days ​

KQL — Access to a specific secret ​

KQL — Failed access attempts (unauthorized or denied) ​

KQL — Access by a specific identity (use OID for managed identities) ​

KQL — Secret access volume by caller (detect anomalous access patterns) ​

9. What Lives in Key Vault vs. What Is a Managed Identity ​

Use Managed Identity For ​

Use Key Vault For ​

What Never Goes Anywhere Except Key Vault ​

Quick Reference Decision Tree ​

Quick Reference ​

Key Vault Secrets Workflow

1. Creating a New Secret

Portal

CLI

2. Granting Access

Managed Identity (preferred — runtime access from Container Apps)

Service Principal (for automation, CI/CD, or cross-tenant access)

RBAC Roles Reference

3. Referencing a Secret in Terraform

4. Referencing a Secret in Container Apps

Method A: secretsRef (Recommended)

Method B: Direct SDK call in Python (runtime fetch)

Triggering a new revision after secret update

5. GitHub Actions — Why Secrets Are Not Stored There

6. Secret Rotation Procedure

Step 1: Generate the new secret value

Step 2: Create a new version in Key Vault

Step 3: Force apps to pick up the new version

Step 4: Verify the app is working with the new secret

Step 5: Disable the old version

Step 6: Revoke the old credential at the source

7. Emergency Rotation

Immediate steps (target: under 15 minutes)

Scope the blast radius

Follow-up (within 24 hours)

8. Audit — KQL Query for Key Vault Access

KQL — All secret operations in the last 7 days

KQL — Access to a specific secret

KQL — Failed access attempts (unauthorized or denied)

KQL — Access by a specific identity (use OID for managed identities)

KQL — Secret access volume by caller (detect anomalous access patterns)

9. What Lives in Key Vault vs. What Is a Managed Identity

Use Managed Identity For

Use Key Vault For

What Never Goes Anywhere Except Key Vault

Quick Reference Decision Tree

Quick Reference