Version: Next

Monitor Validation Progress

DOCUMENT CATEGORY: Reference SCOPE: Phase 05 Task 9 — cluster validation monitoring PURPOSE: Real-time dashboard combining Azure API step status with live EnvironmentValidatorFull log streaming from cluster nodes RUN DURING: Task 9: Validation — after clicking Start validation in the portal

Status: Active Estimated Runtime: 15–20 minutes (matches portal validation duration) Last Updated: 2026-03-09

Overview

Runs a while ($true) dashboard loop that refreshes every 10 seconds. Displays:

All validation step names with pass/fail/in-progress/pending/skipped status
Per-step duration (start → end timestamps from Azure API)
Progress bar showing completed vs total steps
Live tail of EnvironmentValidatorFull.*.log from whichever cluster node has the most recent log file
Error/warning counts from the log file
Node connectivity status (handles domain-join reboots gracefully)

Auto-exits when validation completes (success or failure). Press Ctrl+C to exit manually at any time.

Script

Script: scripts/deploy/04-cluster-deployment/phase-05-cluster-deployment/monitoring/powershell/Monitor-Validation.ps1

Parameters

Parameter	Type	Required	YAML Path	Description
`-ResourceGroupName`	string	Yes	`compute.arm_deployment.cluster_resource_group`	Resource group containing the cluster
`-ClusterName`	string	Yes	`compute.arm_deployment.cluster_name`	Cluster name
`-SubscriptionId`	string	Yes	`azure_platform.azure_tenants[*].aztenant_subscription_id`	Cluster subscription ID
`-NodeIPs`	string[]	Yes	`compute.cluster_nodes[*].management_ip`	Node management IPs for log access
`-LocalAdminUsername`	string	No	`identity.accounts.account_local_admin_username`	Local admin for PSRemoting (tier 1)
`-LocalAdminPassword`	SecureString	No	`identity.accounts.account_local_admin_password`	Local admin password (tier 1)
`-KeyVaultName`	string	No	`security.keyvault.kv_azl.kv_azl_name`	Key Vault for credential resolution (tier 2)
`-KeyVaultSubscriptionId`	string	No	`azure_platform.azure_tenants[*].aztenant_subscription_id`	Subscription where KV resides
`-TenantId`	string	No	`azure_platform.azure_tenants[*].aztenant_id`	Tenant ID for SPN auth
`-SPNClientId`	string	No	—	SPN app ID for Azure auth (tier 1)
`-SPNClientSecret`	SecureString	No	—	SPN secret (tier 1)
`-LogBasePath`	string	No	—	Log base path on nodes (default: `C:\CloudDeployment\Logs`)
`-RefreshInterval`	int	No	—	Dashboard refresh in seconds (default: 10)
`-LogTailLines`	int	No	—	Log lines to display (default: 15)

Usage

Start validation monitor — minimum required params
.\Monitor-Validation.ps1 `
 -ResourceGroupName "rg-iic-azurelocal-prod" `
 -ClusterName "iic-clus01" `
 -SubscriptionId "<SUBSCRIPTION_ID>" `
 -NodeIPs @("10.10.1.11", "10.10.1.12", "10.10.1.13") `
 -KeyVaultName "kv-iic-platform"

Direct credentials — no Key Vault
$pwd = Read-Host -AsSecureString "Local admin password"
.\Monitor-Validation.ps1 `
 -ResourceGroupName "rg-iic-azurelocal-prod" `
 -ClusterName "iic-clus01" `
 -SubscriptionId "<SUBSCRIPTION_ID>" `
 -NodeIPs @("10.10.1.11", "10.10.1.12") `
 -LocalAdminUsername "iic-localadmin" `
 -LocalAdminPassword $pwd `
 -KeyVaultName ""

Faster refresh, more log lines
.\Monitor-Validation.ps1 `
 -ResourceGroupName "rg-iic-azurelocal-prod" `
 -ClusterName "iic-clus01" `
 -SubscriptionId "<SUBSCRIPTION_ID>" `
 -NodeIPs @("10.10.1.11", "10.10.1.12") `
 -KeyVaultName "kv-iic-platform" `
 -RefreshInterval 5 `
 -LogTailLines 20

Full Script

Monitor-Validation.ps1
#Requires -Version 7.0

<#
.SYNOPSIS
 Monitor-Validation.ps1
 Real-time dashboard for Azure Local cluster validation monitoring with log file streaming.

.DESCRIPTION
 Provides a full-screen dashboard view during portal validation (Task 9):
 - Validation step progress and status from Azure API (deploymentSettings/default)
 - Real-time tail of EnvironmentValidatorFull log from cluster nodes via PSRemoting
 - Per-step duration tracking
 - Node connectivity status (handles reboots gracefully)
 - Auto-exits on validation completion (success or failure)

 Run after clicking "Start validation" in the Azure Portal.
 Press Ctrl+C to exit at any time.

 Credential Resolution Order (node access):
 1. -LocalAdminUsername + -LocalAdminPassword parameters
 2. Key Vault: -KeyVaultName with username/password secret names
 3. Interactive Read-Host prompt

 Azure Authentication Order:
 1. -SPNClientId + -SPNClientSecret parameters
 2. Key Vault: SPN credentials via -SPNClientIdSecretName / -SPNClientSecretSecretName
 3. Existing Connect-AzAccount / Get-AzContext

.PARAMETER ResourceGroupName
 Resource group containing the cluster. (compute.arm_deployment.cluster_resource_group)

.PARAMETER ClusterName
 Name of the Azure Local cluster. (compute.arm_deployment.cluster_name)

.PARAMETER SubscriptionId
 Azure subscription ID for the cluster. (azure_platform.azure_tenants[*].aztenant_subscription_id)

.PARAMETER NodeIPs
 Array of node management IP addresses for PSRemoting log access. (compute.cluster_nodes[*].management_ip)

.PARAMETER LocalAdminUsername
 Local admin username for PSRemoting. If omitted, resolved from Key Vault or prompted.
 (identity.accounts.account_local_admin_username)

.PARAMETER LocalAdminPassword
 Local admin password as SecureString. If omitted, resolved from Key Vault or prompted.
 (identity.accounts.account_local_admin_password)

.PARAMETER KeyVaultName
 Key Vault name for credential resolution. Set to empty string "" to skip KV lookup.
 (security.keyvault.kv_azl.kv_azl_name)

.PARAMETER KeyVaultSubscriptionId
 Subscription ID where the Key Vault resides. Defaults to SubscriptionId if not set.

.PARAMETER TenantId
 Azure AD tenant ID for SPN authentication. (azure_platform.azure_tenants[*].aztenant_id)

.PARAMETER SPNClientId
 Service Principal app/client ID. If omitted, retrieved from Key Vault.

.PARAMETER SPNClientSecret
 Service Principal client secret (SecureString). If omitted, retrieved from Key Vault.

.PARAMETER SPNClientIdSecretName
 Key Vault secret name for the SPN client ID (default: sp-azurelocal-client-id).

.PARAMETER SPNClientSecretSecretName
 Key Vault secret name for the SPN client secret (default: sp-azurelocal-client-secret).

.PARAMETER LocalAdminUsernameSecretName
 Key Vault secret name for the local admin username (default: local-admin-username).

.PARAMETER LocalAdminPasswordSecretName
 Key Vault secret name for the local admin password (default: local-admin-password).

.PARAMETER LogBasePath
 Base path where CloudDeployment logs are stored on nodes (default: C:\CloudDeployment\Logs).

.PARAMETER RefreshInterval
 Seconds between dashboard refreshes (default: 10).

.PARAMETER LogTailLines
 Number of recent log lines to display (default: 15).

.EXAMPLE
 .\Monitor-Validation.ps1 `
 -ResourceGroupName "rg-iic-azurelocal-prod" `
 -ClusterName "iic-clus01" `
 -SubscriptionId "<SUBSCRIPTION_ID>" `
 -NodeIPs @("10.10.1.11","10.10.1.12","10.10.1.13") `
 -KeyVaultName "kv-iic-platform"

.EXAMPLE
 $pwd = Read-Host -AsSecureString 'Password'
 .\Monitor-Validation.ps1 `
 -ResourceGroupName "rg-iic-azurelocal-prod" `
 -ClusterName "iic-clus01" `
 -SubscriptionId "<SUBSCRIPTION_ID>" `
 -NodeIPs @("10.10.1.11","10.10.1.12") `
 -LocalAdminUsername "iic-localadmin" `
 -LocalAdminPassword $pwd `
 -KeyVaultName ""

.NOTES
 Author: Azure Local Cloud Azure Local Cloud
 Version: 1.0.0
 Phase: 05-cluster-deployment
 Task: Task 9 — Validation (run concurrently with portal validation)
 Execution: Run from any machine with PS 7+, Az module, and network access to node IPs
 Requires: PowerShell 7+, Az module, Azure CLI
 Run after: Clicking "Start validation" in Azure Portal
 Source: temp/Monitor-Validation-Enhanced.ps1
#>

[CmdletBinding()]
param(
 # --- Cluster / Environment ---
 [Parameter(Mandatory = $true)]
 [string]$ResourceGroupName, # compute.arm_deployment.cluster_resource_group

 [Parameter(Mandatory = $true)]
 [string]$ClusterName, # compute.arm_deployment.cluster_name

 [Parameter(Mandatory = $true)]
 [string]$SubscriptionId, # azure_platform.azure_tenants[*].aztenant_subscription_id

 [Parameter(Mandatory = $true)]
 [string[]]$NodeIPs, # compute.cluster_nodes[*].management_ip

 # --- Node credentials (direct — tier 1) ---
 [Parameter(Mandatory = $false)]
 [string]$LocalAdminUsername, # identity.accounts.account_local_admin_username

 [Parameter(Mandatory = $false)]
 [securestring]$LocalAdminPassword,

 # --- Key Vault (tier 2) ---
 [Parameter(Mandatory = $false)]
 [string]$KeyVaultName = "", # security.keyvault.kv_azl.kv_azl_name

 [Parameter(Mandatory = $false)]
 [string]$KeyVaultSubscriptionId = "", # defaults to SubscriptionId

 [Parameter(Mandatory = $false)]
 [string]$LocalAdminUsernameSecretName = "local-admin-username",

 [Parameter(Mandatory = $false)]
 [string]$LocalAdminPasswordSecretName = "local-admin-password",

 # --- SPN Authentication ---
 [Parameter(Mandatory = $false)]
 [string]$TenantId = "", # azure_platform.azure_tenants[*].aztenant_id

 [Parameter(Mandatory = $false)]
 [string]$SPNClientId,

 [Parameter(Mandatory = $false)]
 [securestring]$SPNClientSecret,

 [Parameter(Mandatory = $false)]
 [string]$SPNClientIdSecretName = "sp-azurelocal-client-id",

 [Parameter(Mandatory = $false)]
 [string]$SPNClientSecretSecretName = "sp-azurelocal-client-secret",

 # --- Display ---
 [Parameter(Mandatory = $false)]
 [string]$LogBasePath = "C:\CloudDeployment\Logs",

 [Parameter(Mandatory = $false)]
 [int]$RefreshInterval = 10,

 [Parameter(Mandatory = $false)]
 [int]$LogTailLines = 15
)

$ErrorActionPreference = 'SilentlyContinue'
if (-not $KeyVaultSubscriptionId) { $KeyVaultSubscriptionId = $SubscriptionId }

# =============================================================================
# HELPER FUNCTIONS
# =============================================================================

function Test-NodeConnectivity {
 param([string]$NodeIP, [int]$TimeoutSeconds = 5)
 try { return Test-Connection -ComputerName $NodeIP -Count 1 -Quiet -TimeoutSeconds $TimeoutSeconds -ErrorAction Stop }
 catch { return $false }
}

function Get-NodeCredential {
 param(
 [string]$Username, [securestring]$Password,
 [string]$KeyVaultName, [string]$KeyVaultSubscriptionId,
 [string]$UsernameSecret, [string]$PasswordSecret
 )
 # Tier 1: Direct parameters
 if ($Username -and $Password) {
 Write-Host " [INFO] Using credentials supplied via parameters" -ForegroundColor Cyan
 return New-Object System.Management.Automation.PSCredential($Username, $Password)
 }
 # Tier 2: Key Vault
 if ($KeyVaultName) {
 try {
 Write-Host " [INFO] Retrieving credentials from Key Vault '$KeyVaultName'..." -ForegroundColor Cyan
 $ctx = Get-AzContext
 if ($ctx.Subscription.Id -ne $KeyVaultSubscriptionId) {
 Set-AzContext -SubscriptionId $KeyVaultSubscriptionId -ErrorAction Stop | Out-Null
 }
 $kvUser = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name $UsernameSecret -AsPlainText -ErrorAction Stop
 $kvPass = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name $PasswordSecret -AsPlainText -ErrorAction Stop
 if ($ctx.Subscription.Id -ne $KeyVaultSubscriptionId) {
 Set-AzContext -SubscriptionId $ctx.Subscription.Id -ErrorAction SilentlyContinue | Out-Null
 }
 if ($kvUser -and $kvPass) {
 Write-Host " [OK] Credentials retrieved from Key Vault" -ForegroundColor Green
 return New-Object System.Management.Automation.PSCredential($kvUser, (ConvertTo-SecureString $kvPass -AsPlainText -Force))
 }
 } catch {
 Write-Host " [WARN] Key Vault lookup failed: $($_.Exception.Message)" -ForegroundColor Yellow
 }
 }
 # Tier 3: Interactive
 Write-Host " Please enter local admin credentials for node access:" -ForegroundColor Yellow
 $promptUser = Read-Host -Prompt " Local Admin Username"
 $promptPass = Read-Host -Prompt " Local Admin Password" -AsSecureString
 if (-not $promptUser -or -not $promptPass) {
 Write-Warning "No credentials provided. Log monitoring will be disabled."
 return $null
 }
 return New-Object System.Management.Automation.PSCredential($promptUser, $promptPass)
}

function Get-LatestLogFile {
 param([string]$BasePath, [string[]]$NodeIPs, [System.Management.Automation.PSCredential]$Credential, [ref]$NodeStatus)
 $allLogs = @()
 foreach ($nodeIP in $NodeIPs) {
 if (-not (Test-NodeConnectivity -NodeIP $nodeIP -TimeoutSeconds 3)) {
 if ($NodeStatus) { $NodeStatus.Value[$nodeIP] = "Offline (may be rebooting)" }
 continue
 }
 try {
 $logFiles = Invoke-Command -ComputerName $nodeIP -Credential $Credential -ScriptBlock {
 param($Path)
 if (Test-Path $Path) {
 Get-ChildItem -Path $Path -Filter "EnvironmentValidatorFull.*.log" -ErrorAction SilentlyContinue |
 Sort-Object LastWriteTime -Descending | Select-Object -First 1
 }
 } -ArgumentList $BasePath -ErrorAction Stop
 if ($NodeStatus) { $NodeStatus.Value[$nodeIP] = "Online" }
 if ($logFiles) {
 $allLogs += @{ NodeIP = $nodeIP; FullName = $logFiles.FullName; LastWriteTime = $logFiles.LastWriteTime; Length = $logFiles.Length }
 }
 } catch {
 if ($NodeStatus) {
 $NodeStatus.Value[$nodeIP] = if ($_.Exception.Message -match "WinRM|cannot be contacted|not respond") { "WinRM unavailable (rebooting?)" } else { "Error: $($_.Exception.Message)" }
 }
 }
 }
 if ($allLogs.Count -gt 0) { return $allLogs | Sort-Object LastWriteTime -Descending | Select-Object -First 1 }
 return $null
}

function Get-LogTail {
 param([hashtable]$LogFileInfo, [int]$Lines = 15, [System.Management.Automation.PSCredential]$Credential)
 if (-not $LogFileInfo) { return @() }
 try {
 return Invoke-Command -ComputerName $LogFileInfo.NodeIP -Credential $Credential -ScriptBlock {
 param($LogPath, $NumLines)
 if (Test-Path $LogPath) { Get-Content -Path $LogPath -Tail $NumLines -ErrorAction SilentlyContinue }
 } -ArgumentList $LogFileInfo.FullName, $Lines -ErrorAction Stop
 } catch { return @() }
}

function Get-LogSummary {
 param([hashtable]$LogFileInfo, [System.Management.Automation.PSCredential]$Credential)
 if (-not $LogFileInfo) { return @{ Errors = 0; Warnings = 0; Info = 0; LastModified = "N/A"; Size = "N/A" } }
 try {
 return Invoke-Command -ComputerName $LogFileInfo.NodeIP -Credential $Credential -ScriptBlock {
 param($Path)
 if (Test-Path $Path) {
 $content = Get-Content -Path $Path -ErrorAction Stop
 $fi = Get-Item $Path
 @{
 Errors = ($content | Select-String -Pattern "\[Error\]|\[ERR\]|ERROR:" -AllMatches).Count
 Warnings = ($content | Select-String -Pattern "\[Warning\]|\[WARN\]|WARNING:" -AllMatches).Count
 Info = ($content | Select-String -Pattern "\[Info\]|\[INF\]|INFO:" -AllMatches).Count
 LastModified = $fi.LastWriteTime.ToString("HH:mm:ss")
 Size = "{0:N2} MB" -f ($fi.Length / 1MB)
 }
 }
 } -ArgumentList $LogFileInfo.FullName -ErrorAction Stop
 } catch { return @{ Errors = 0; Warnings = 0; Info = 0; LastModified = "N/A"; Size = "N/A" } }
}

function Get-ValidationStatus {
 param([string]$SubscriptionId, [string]$ResourceGroupName, [string]$ClusterName)
 $uri = "https://management.azure.com/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroupName/providers/microsoft.azurestackhci/clusters/$ClusterName/deploymentSettings/default?api-version=2024-04-01"
 try { return (az rest --method get --uri $uri 2>$null | ConvertFrom-Json).properties.reportedProperties.validationStatus }
 catch { return $null }
}

function Get-ClusterStatus {
 param([string]$SubscriptionId, [string]$ResourceGroupName, [string]$ClusterName)
 try {
 $cluster = Get-AzResource -ResourceGroupName $ResourceGroupName -ResourceType "Microsoft.AzureStackHCI/clusters" -Name $ClusterName -ExpandProperties -ErrorAction SilentlyContinue
 return $cluster.Properties.status
 } catch { return "Unknown" }
}

function Write-ProgressBar {
 param([int]$Completed, [int]$Total, [int]$Width = 40)
 $percent = if ($Total -gt 0) { [math]::Round(($Completed / $Total) * 100) } else { 0 }
 $filled = [math]::Round(($percent / 100) * $Width)
 $empty = $Width - $filled
 Write-Host " [" -NoNewline -ForegroundColor Gray
 Write-Host ("█" * $filled) -NoNewline -ForegroundColor Green
 Write-Host ("░" * $empty) -NoNewline -ForegroundColor DarkGray
 Write-Host "] " -NoNewline -ForegroundColor Gray
 Write-Host "$percent%" -NoNewline -ForegroundColor Cyan
 Write-Host " ($Completed/$Total steps)" -ForegroundColor Gray
}

# =============================================================================
# AZURE AUTHENTICATION (Parameters -> Key Vault -> Current Context)
# =============================================================================
Write-Host "`n Starting Azure Local Validation Monitor..." -ForegroundColor Cyan
Write-Host " Monitoring: $ClusterName in $ResourceGroupName" -ForegroundColor Gray
Write-Host " Log nodes: $($NodeIPs -join ', ')" -ForegroundColor Gray
Write-Host ""

Write-Host " Authenticating to Azure..." -ForegroundColor Yellow
$spnAuthenticated = $false

if ($SPNClientId -and $SPNClientSecret) {
 try {
 $spnCred = New-Object System.Management.Automation.PSCredential($SPNClientId, $SPNClientSecret)
 Connect-AzAccount -ServicePrincipal -Credential $spnCred -Tenant $TenantId -Subscription $KeyVaultSubscriptionId -ErrorAction Stop | Out-Null
 Write-Host " [OK] Authenticated via SPN (parameters)" -ForegroundColor Green
 $spnAuthenticated = $true
 } catch { Write-Host " [WARN] SPN parameter auth failed: $($_.Exception.Message)" -ForegroundColor Yellow }
}

if (-not $spnAuthenticated -and $KeyVaultName) {
 try {
 $spnAppId = az keyvault secret show --vault-name $KeyVaultName --name $SPNClientIdSecretName --query "value" -o tsv --subscription $KeyVaultSubscriptionId 2>$null
 $spnSecretVal = az keyvault secret show --vault-name $KeyVaultName --name $SPNClientSecretSecretName --query "value" -o tsv --subscription $KeyVaultSubscriptionId 2>$null
 if ($spnAppId -and $spnSecretVal) {
 $spnCred = New-Object System.Management.Automation.PSCredential($spnAppId, (ConvertTo-SecureString $spnSecretVal -AsPlainText -Force))
 Connect-AzAccount -ServicePrincipal -Credential $spnCred -Tenant $TenantId -Subscription $KeyVaultSubscriptionId -ErrorAction Stop | Out-Null
 Write-Host " [OK] Authenticated via SPN (Key Vault)" -ForegroundColor Green
 $spnAuthenticated = $true
 }
 } catch { Write-Host " [WARN] SPN Key Vault auth failed: $($_.Exception.Message)" -ForegroundColor Yellow }
}

if (-not $spnAuthenticated) {
 $ctx = Get-AzContext -ErrorAction SilentlyContinue
 if ($ctx) { Write-Host " [INFO] Using existing Azure context: $($ctx.Account.Id)" -ForegroundColor Green }
 else { Write-Host " [ERROR] Not authenticated to Azure. Run Connect-AzAccount first." -ForegroundColor Red; exit 1 }
}

Write-Host " Resolving node credentials..." -ForegroundColor Yellow
$nodeCredential = Get-NodeCredential `
 -Username $LocalAdminUsername -Password $LocalAdminPassword `
 -KeyVaultName $KeyVaultName -KeyVaultSubscriptionId $KeyVaultSubscriptionId `
 -UsernameSecret $LocalAdminUsernameSecretName -PasswordSecret $LocalAdminPasswordSecretName

if (-not $nodeCredential) {
 Write-Host " [WARN] No node credentials — log monitoring will be disabled." -ForegroundColor Yellow
} else {
 Write-Host " [OK] Node credentials resolved" -ForegroundColor Green
}

Set-AzContext -Subscription $SubscriptionId -ErrorAction SilentlyContinue | Out-Null
Write-Host ""; Start-Sleep -Seconds 2

$startTime = Get-Date
$nodeStatus = @{}

# =============================================================================
# DASHBOARD LOOP
# =============================================================================
while ($true) {
 $validation = Get-ValidationStatus -SubscriptionId $SubscriptionId -ResourceGroupName $ResourceGroupName -ClusterName $ClusterName
 $clusterStatus = Get-ClusterStatus -SubscriptionId $SubscriptionId -ResourceGroupName $ResourceGroupName -ClusterName $ClusterName
 $logFileInfo = $null
 if ($nodeCredential) {
 $logFileInfo = Get-LatestLogFile -BasePath $LogBasePath -NodeIPs $NodeIPs -Credential $nodeCredential -NodeStatus ([ref]$nodeStatus)
 }

 Clear-Host
 $elapsed = (Get-Date) - $startTime
 $elapsedStr = "{0:hh\:mm\:ss}" -f $elapsed

 # Header
 Write-Host ""
 Write-Host " ╔══════════════════════════════════════════════════════════════════════════╗" -ForegroundColor Cyan
 Write-Host " ║ AZURE LOCAL VALIDATION DASHBOARD + LOGS ║" -ForegroundColor Cyan
 Write-Host " ╚══════════════════════════════════════════════════════════════════════════╝" -ForegroundColor Cyan
 Write-Host ""

 $statusColor = switch ($clusterStatus) {
 "ValidationInProgress" { "Yellow" }; "DeploymentInProgress" { "Cyan" }
 "Succeeded" { "Green" }; "Failed" { "Red" }
 default { "White" }
 }
 Write-Host " Cluster: " -NoNewline -ForegroundColor Gray
 Write-Host "$ClusterName" -NoNewline -ForegroundColor White
 Write-Host " | Status: " -NoNewline -ForegroundColor Gray
 Write-Host "$clusterStatus" -NoNewline -ForegroundColor $statusColor
 Write-Host " | Elapsed: " -NoNewline -ForegroundColor Gray
 Write-Host "$elapsedStr" -ForegroundColor Yellow
 Write-Host ""

 # Validation steps
 if ($validation -and $validation.steps) {
 $steps = $validation.steps
 $successCount = ($steps | Where-Object { $_.status -eq "Success" }).Count
 $skippedCount = ($steps | Where-Object { $_.status -eq "Skipped" }).Count
 $failedCount = ($steps | Where-Object { $_.status -eq "Error" }).Count
 $inProgCount = ($steps | Where-Object { $_.status -eq "InProgress" }).Count
 $pendingCount = ($steps | Where-Object { -not $_.status -or $_.status -notin @("Success","Error","InProgress","Skipped") }).Count
 $totalSteps = $steps.Count
 $completedCount = $successCount + $skippedCount

 Write-ProgressBar -Completed $completedCount -Total $totalSteps
 Write-Host ""

 Write-Host " ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐" -ForegroundColor DarkGray
 Write-Host " │ VALIDATION STEPS │" -ForegroundColor DarkGray
 Write-Host " ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤" -ForegroundColor DarkGray

 foreach ($step in $steps) {
 $iconChar = switch ($step.status) {
 "Success" { "[OK]"; $color = "Green" }
 "Skipped" { "[--]"; $color = "DarkCyan" }
 "Error" { "[XX]"; $color = "Red" }
 "InProgress" { "[..]"; $color = "Yellow" }
 default { "[ ]"; $color = "DarkGray" }
 }
 $name = ($step.name -replace "Azure Stack HCI |Azure Stack ","")
 if ($name.Length -gt 55) { $name = $name.Substring(0,52) + "..." }
 $name = $name.PadRight(55)
 $duration = ""
 if ($step.startTimeUtc -and $step.startTimeUtc -ne "NA") {
 if ($step.endTimeUtc -and $step.endTimeUtc -ne "NA") {
 $duration = "{0:mm\:ss}" -f ([DateTime]::Parse($step.endTimeUtc) - [DateTime]::Parse($step.startTimeUtc))
 } elseif ($step.status -eq "InProgress") {
 $duration = "{0:mm\:ss}" -f ((Get-Date).ToUniversalTime() - [DateTime]::Parse($step.startTimeUtc))
 }
 }
 Write-Host " │ " -NoNewline -ForegroundColor DarkGray
 Write-Host $iconChar -NoNewline -ForegroundColor $color
 Write-Host " $name $($duration.PadLeft(6))" -NoNewline -ForegroundColor White
 Write-Host " │" -ForegroundColor DarkGray
 }

 Write-Host " └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘" -ForegroundColor DarkGray
 Write-Host ""

 Write-Host " Summary: $successCount Success" -NoNewline -ForegroundColor Green
 if ($skippedCount -gt 0) { Write-Host " | $skippedCount Skipped" -NoNewline -ForegroundColor DarkCyan }
 if ($failedCount -gt 0) { Write-Host " | $failedCount Failed" -NoNewline -ForegroundColor Red }
 if ($inProgCount -gt 0) { Write-Host " | $inProgCount Running" -NoNewline -ForegroundColor Yellow }
 if ($pendingCount -gt 0) { Write-Host " | $pendingCount Pending" -NoNewline -ForegroundColor DarkGray }
 Write-Host ""
 Write-Host ""

 if ($validation.status -eq "Succeeded" -or ($completedCount -eq $totalSteps -and $failedCount -eq 0)) {
 Write-Host " ╔══════════════════════════════════════════════════════════════════════════╗" -ForegroundColor Green
 Write-Host " ║ [OK] VALIDATION COMPLETED SUCCESSFULLY ║" -ForegroundColor Green
 Write-Host " ╚══════════════════════════════════════════════════════════════════════════╝" -ForegroundColor Green
 Write-Host ""
 } elseif ($validation.status -eq "Failed" -or $failedCount -gt 0) {
 Write-Host " ╔══════════════════════════════════════════════════════════════════════════╗" -ForegroundColor Red
 Write-Host " ║ [XX] VALIDATION FAILED ║" -ForegroundColor Red
 Write-Host " ╚══════════════════════════════════════════════════════════════════════════╝" -ForegroundColor Red
 Write-Host ""
 }
 } else {
 Write-Host " Waiting for validation data from Azure API..." -ForegroundColor Yellow
 Write-Host ""
 }

 # Log section
 Write-Host " ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐" -ForegroundColor Magenta
 Write-Host " │ VALIDATION LOG — EnvironmentValidatorFull │" -ForegroundColor Magenta
 Write-Host " ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤" -ForegroundColor Magenta

 if ($nodeCredential -and $nodeStatus.Count -gt 0) {
 foreach ($ip in $NodeIPs) {
 $st = if ($nodeStatus.ContainsKey($ip)) { $nodeStatus[$ip] } else { "Unknown" }
 $activeLog = ($logFileInfo -and $logFileInfo.NodeIP -eq $ip)
 $display = if ($activeLog) { "$st (live logs)" } else { $st }
 $stColor = if ($st -eq "Online") { "Green" } elseif ($st -match "Offline|unavailable|rebooting") { "Yellow" } else { "Red" }
 Write-Host " │ $ip - " -NoNewline -ForegroundColor DarkGray
 Write-Host $display -ForegroundColor $stColor
 }
 Write-Host " ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤" -ForegroundColor Magenta
 }

 if ($logFileInfo) {
 $summary = Get-LogSummary -LogFileInfo $logFileInfo -Credential $nodeCredential
 $logName = Split-Path $logFileInfo.FullName -Leaf
 Write-Host " │ $($logFileInfo.NodeIP) | $logName" -ForegroundColor DarkGray
 Write-Host " │ Size: $($summary.Size) Modified: $($summary.LastModified) " -NoNewline -ForegroundColor DarkGray
 if ($summary.Errors -gt 0) { Write-Host "E:$($summary.Errors) " -NoNewline -ForegroundColor Red }
 if ($summary.Warnings -gt 0) { Write-Host "W:$($summary.Warnings) " -NoNewline -ForegroundColor Yellow }
 Write-Host ""
 Write-Host " ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤" -ForegroundColor Magenta

 $logLines = Get-LogTail -LogFileInfo $logFileInfo -Lines $LogTailLines -Credential $nodeCredential
 if (-not $logLines -or $logLines.Count -eq 0) {
 Write-Host " │ No log content available" -ForegroundColor Yellow
 } else {
 foreach ($line in $logLines) {
 if ([string]::IsNullOrWhiteSpace($line)) { Write-Host " │"; continue }
 if ($line -match '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ') { $line = $line -replace '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ','' }
 if ($line.Length -gt 120) { $line = $line.Substring(0,117) + "..." }
 $lineColor = if ($line -match "\[Error\]|\[ERR\]|ERROR:|Failed|Exception") { "Red" }
 elseif ($line -match "\[Warning\]|\[WARN\]|WARNING:|Retry") { "Yellow" }
 elseif ($line -match "\[Info\]|\[INF\]|INFO:") { "Cyan" }
 elseif ($line -match "Success|Succeeded|Complete|Passed") { "Green" }
 elseif ($line -match "Verbose") { "DarkGray" }
 else { "Gray" }
 Write-Host " │ $line" -ForegroundColor $lineColor
 }
 }
 } elseif (-not $nodeCredential) {
 Write-Host " │ Log monitoring disabled — no credentials available" -ForegroundColor Yellow
 } else {
 Write-Host " │ No EnvironmentValidatorFull log found on: $($NodeIPs -join ', ')" -ForegroundColor Yellow
 }

 Write-Host " └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘" -ForegroundColor Magenta
 Write-Host ""
 Write-Host " Last refresh: $(Get-Date -Format 'HH:mm:ss') | Refresh: ${RefreshInterval}s | Ctrl+C to exit" -ForegroundColor DarkGray

 # Auto-exit on completion
 if ($validation -and $validation.steps) {
 $steps = $validation.steps
 $successCount = ($steps | Where-Object { $_.status -eq "Success" }).Count
 $skippedCount = ($steps | Where-Object { $_.status -eq "Skipped" }).Count
 $failedCount = ($steps | Where-Object { $_.status -eq "Error" }).Count
 $completedCount = $successCount + $skippedCount
 if ($validation.status -eq "Succeeded" -or ($completedCount -eq $steps.Count -and $failedCount -eq 0)) {
 Write-Host ""
 Write-Host " Validation complete. Proceed to Task 10: Review + Create." -ForegroundColor Green
 break
 } elseif ($validation.status -eq "Failed" -or $failedCount -gt 0) {
 Write-Host ""
 Write-Host " Validation failed. Check Azure Portal and log files for details." -ForegroundColor Red
 break
 }
 }

 Start-Sleep -Seconds $RefreshInterval
}

Troubleshooting

Issue	Cause	Resolution
Validation script reports `FAIL` on environment checks	Prerequisites not met before deployment started	Review the specific check that failed; address the prerequisite (e.g., DNS, NTP, AD connectivity) and re-run validation
Script timeout waiting for validation completion	Deployment validation takes longer than expected	Increase `$RefreshInterval`; check Azure portal deployment status for progress; review `C:\CloudDeployment\Logs\` for errors
All validation checks show `Unknown` status	Cluster nodes unreachable or validation service not running	Verify node connectivity: `Test-Connection <node-ip>`; check ECE agent status: `Get-Service LifeCycleManagementAgent`

Version Control

Created: 2026-03-09 by Azure Local Cloudnology Team
Last Updated: 2026-03-09 by Azure Local Cloudnology Team
Version: 1.0.0
Tags: azure-local, monitoring, validation, runbook

Overview​

Script​

Parameters​

Usage​

Full Script​

Troubleshooting​