Docs

Appearance

Runbook: BIZFTPAZP01 — IPS_cirius-admin Stale Credentials ​

Story: DECOMM-019 (also tracked as GitHub Issue #555) System: BIZFTPAZP01 (FTP server, PROD) Account: FTPBIZAP21\IPS_cirius-admin (local-to-server account) Severity: Medium — service is running but the credential tied to the scheduled task is invalid Executor: Kevin or Greg (T1 Domain Admin — server-side AD action required)

Purpose ​

The scheduled task on BIZFTPAZP01 that runs as FTPBIZAP21\IPS_cirius-admin has stale or expired credentials. The linked account either no longer exists or has been disabled / rotated without updating the task credential. This runbook documents what to verify, how to rotate, and how to confirm recovery.

Kobe cannot execute this directly — the operation requires a signed-in RDP session with T1 domain admin rights on the FTP server. Kevin or Greg run this. Kobe opens the PR for this runbook and updates the story when they confirm execution.

What BIZFTPAZP01 Is ​

BIZFTPAZP01 is the primary PROD FTP server. It runs Windows Server with the FTP role under IIS (port 21, passive range configured on the Palo Alto edge firewall) and accepts inbound FTP from external trading partners for clinical file drops. The server is in the dmz resource group, attached to the DMZ VNet, and fronted by the Palo Alto edge firewall.

The FTP service itself runs under an IIS application pool identity. The IPS_cirius-admin account is used by a separate scheduled task on the server — it is not the FTP service account. The scheduled task handles file archival / move operations after successful uploads (promoting files from the inbound drop zone to the processing directory where downstream systems pick them up).

Terraform: azure-infra/dmz/svr_bizftp.tf (VM), azure-infra/prod/svr_bizftp.tf (there is a second BIZFTPAZP01 in prod — confirm which one this issue references; the story text says FTPBIZAP21\IPS_cirius-admin which suggests the FTP server is named FTPBIZAP21 in the local SAM database despite the Azure VM name).

The Problem ​

Windows Event Viewer on BIZFTPAZP01 shows Task Scheduler errors similar to:

Task Scheduler failed to log on '\IPS_cirius-admin' .
Failure occurred in 'LogonUserExEx'.
User Action: Ensure the credentials for the task are correctly specified.
Additional Data: Error Value: 2147943726.

0x8007052E / 2147943726 = ERROR_LOGON_FAILURE — bad username or password.

Root causes, in order of likelihood:

  1. Local account IPS_cirius-admin password was rotated (or the account was recreated) but the scheduled task Run as: credential was not updated
  2. Account was disabled / removed and the scheduled task still references it
  3. Password policy forced expiry and no one logged in interactively to rotate

Until this is fixed, the archival task fails on every run. Files accumulate in the inbound drop zone and processing downstream is delayed. There is no known data-loss risk because uploads still succeed — only the post-upload move is broken.

Pre-flight Checks (Kobe can run these read-only) ​

All checks run from the Cloud PC or an admin workstation with Azure access.

1. Confirm the server is up ​

az vm show -g rg-dmz -n BIZFTPAZP01 --show-details --query "{name:name, powerState:powerState}" -o table

Expected: VM running.

2. Check recent task scheduler failures in LAW ​

kusto
Event
| where Computer startswith "BIZFTPAZP01" or Computer startswith "FTPBIZAP21"
| where EventLog == "Microsoft-Windows-TaskScheduler/Operational"
| where RenderedDescription contains "IPS_cirius-admin"
| project TimeGenerated, EventID, Computer, RenderedDescription
| order by TimeGenerated desc
| take 50

Expected: EventID 101 (task logon failure) or 103 (failed to start) entries referencing IPS_cirius-admin. Note the failure cadence — the failure interval is the task's schedule, which helps time the rotation window.

3. Confirm FTP service itself is healthy ​

kusto
Event
| where Computer startswith "BIZFTPAZP01" or Computer startswith "FTPBIZAP21"
| where EventLog == "System"
| where Source == "Microsoft-Windows-IIS-Logging" or Source == "FTP"
| where TimeGenerated > ago(24h)
| project TimeGenerated, EventID, RenderedDescription
| order by TimeGenerated desc
| take 50

Expected: no service-stopped events. FTP uploads should still be completing — the archival task is separate from the FTP listener.

4. Verify the account still exists (AD query only) ​

From any domain-joined admin workstation:

net user IPS_cirius-admin /domain 2>&1

If the account is domain-based. If local-only, connect to the server and run net user IPS_cirius-admin locally (requires RDP, so Kevin/Greg territory).

Remediation (Kevin or Greg executes) ​

Option A: Account still exists, password rotated ​

This is the most common case.

  1. Connect to BIZFTPAZP01 via RDP through Twingate as a T1 domain admin.

  2. Confirm the task name:

    • Open Task Scheduler (taskschd.msc)
    • Navigate to Task Scheduler Library
    • Locate the task configured to run as IPS_cirius-admin (likely in a custom folder — check Cirius, FTP, or root)
    • Note the exact task name and its action (which script/exe, arguments, working dir)

  3. Reset the IPS_cirius-admin password to a new strong password:

    • If local account: Computer Management → Local Users and Groups → Users → right-click IPS_cirius-admin → Set Password → choose a new 32-char random password
    • If domain account: Active Directory Users and Computers → locate the account → Reset Password
    • Record the new password in the Cirius password vault immediately — do not skip this.

  4. Update the scheduled task credential:

    • In Task Scheduler, right-click the task → Properties → General tab
    • Change User or Group... → enter IPS_cirius-admin → OK
    • Check Run whether user is logged on or not
    • Click OK — Windows prompts for the password; enter the new password from step 3

  5. Run the task manually to confirm:

    • Right-click the task → Run
    • Wait for completion → History tab → confirm exit code 0
    • Alternatively, watch for EventID 102 (task completed) in the Microsoft-Windows-TaskScheduler/Operational log

  6. Verify in LAW (can take up to 15 min to propagate):

    kusto
    Event
| where Computer startswith "BIZFTPAZP01" or Computer startswith "FTPBIZAP21"
| where EventLog == "Microsoft-Windows-TaskScheduler/Operational"
| where TimeGenerated > ago(1h)
| project TimeGenerated, EventID, RenderedDescription
| order by TimeGenerated desc

    Expected: EventID 100 (task started) → 102 (task completed), no 101/103 failures.

Option B: Account was deleted ​

  1. Confirm with the team that the task is still needed. If the archive workflow has been superseded, disable the task rather than recreating the account.

  2. If still needed, recreate the local service account:

    • Computer Management → Local Users and Groups → Users → New User
    • Username: IPS_cirius-admin
    • Password: new 32-char random, stored in the vault
    • Uncheck "User must change password at next logon"
    • Check "Password never expires" (service account, per Cirius standard for non-interactive tasks — document this in the vault entry)
    • After creation, add to the same local groups the prior account held (check the password vault entry for the original account for this list — minimum is typically Users; escalate if the task needs broader rights on specific directories, use NTFS ACLs instead of group membership)

  3. Grant "Log on as a batch job" right:

    • secpol.msc → Local Policies → User Rights Assignment → "Log on as a batch job"
    • Add IPS_cirius-admin

  4. Update the scheduled task credential (same as Option A step 4)

  5. Test and verify (same as Option A steps 5–6)

Option C: Task is obsolete ​

If investigation shows the archive task is no longer needed (downstream consumer was retired, etc.):

  1. Confirm with Rory before deleting
  2. Export the task XML for rollback: schtasks /query /tn "<task name>" /xml > C:\backup\<task>-<date>.xml
  3. Delete the task: schtasks /delete /tn "<task name>" /f
  4. Close this runbook referencing the decision

Post-Remediation ​

  1. Update the password vault with the new credential, rotation date, and the name of the admin who performed the rotation.

  2. Update the story with completion notes via:

    POST /api/stories/DECOMM-019/close
{"completion_notes": "<who> rotated IPS_cirius-admin on <date>, task <name> verified running via event 102", "pr_url": "<this-runbook-pr>"}

  3. Close GitHub issue #555 referencing the SecOps story closure.

  4. Schedule a 6-month recurring rotation for this account — add it to the vault's rotation schedule so this does not happen again.

Why Kobe Cannot Execute This ​

  • The action requires interactive RDP to a PROD Windows server (Kobe operates from the Cloud PC with read-only SP credentials — no interactive Windows login rights)
  • Resetting a local account password requires local admin on the server, which Kobe does not have and should not have
  • Task Scheduler credential updates cannot be scripted safely from Terraform — there is no azurerm_* resource for scheduled task credentials, and scripting this via windows-update Custom Script Extension would require embedding the password in pipeline state

Kevin and Greg hold T1 domain admin. They are the escalation path for server-side AD operations per CLAUDE.md.

References ​

  • SecOps story: DECOMM-019
  • GitHub Issue: Cirius-Group-Inc/bedrock-hub#555
  • Terraform: azure-infra/dmz/svr_bizftp.tf (FTP server VM)
  • CMDB entry: BIZFTPAZP01 (system_type=SERVER, criticality=HIGH, phi_adjacent=true)
  • Related runbook: runbooks/common-tasks.md (password rotation general guidance)

Last updated: 2026-04-16 — initial runbook for DECOMM-019

Internal use only — Cirius Group