Task 04: Cluster Operations
DOCUMENT CATEGORY: Implementation Runbook
SCOPE: A running Azure Local cluster
PURPOSE: Read the health, capacity and alerts of the cluster, and make the changes of day 2 (maintenance mode, live migration, volume ownership) safely
MASTER REFERENCE: Suspend and resume Azure Local machines for maintenance
Status: Active
The script connects to the cluster (cluster.name) or to one node with PowerShell remoting and runs the cluster cmdlets there, so the machine that runs it needs no failover clustering tools. It signs in nowhere and asks nothing. HealthCheck, Capacity and Alerts only read; MaintenanceMode, LiveMigrate and Rebalance change the cluster, and each change is behind -WhatIf and -Confirm.
Values from infrastructure.yml
The values for this task come from infrastructure.yml (schema 5). Every key, with its type, default and example, is in the variables reference.
| Key | What the script does with it |
|---|---|
cluster.name | The cluster to connect to and to operate on |
nodes[].name, nodes[].management.ip_address, identity.active_directory.domain_fqdn | -NodeName is matched against them (a name, a fully qualified name or an address). A name that is not in the file is used as it is, with a warning (with -RequireNodeInFile the run stops with exit code 2) |
Execution Options
- PowerShell on a cluster machine
- PowerShell
PowerShell on a cluster machine
When to use: A single question, by hand
Get-ClusterNode; Get-StoragePool -IsPrimordial $false; Get-VirtualDisk; Get-StorageJob; Get-ClusterNetwork
Get-StorageSubSystem -FriendlyName "Clustered*" | Debug-StorageSubSystem # Health Service faults
Get-ClusterNode -Name "<MachineName>" | Get-ClusterGroup # the roles on a machine
Move-ClusterVirtualMachineRole -Name "<VirtualMachineRole>" -Node "<Target>" -MigrationType Live
PowerShell
When to use: The same, with one summary, a JSON file and an exit code
| Script | What it does |
|---|---|
scripts/deploy/08-day2-operations/task-04-cluster-operations/powershell/Invoke-ClusterOperations.ps1 | One action per run. One row per finding or step (Item, Action, Detail) |
| Parameter | Meaning |
|---|---|
-InfrastructureFile | The infrastructure.yml (schema 5). Required |
-Action | HealthCheck, MaintenanceMode, LiveMigrate, Rebalance, Capacity or Alerts. Required |
-NodeName | The machine of MaintenanceMode and LiveMigrate |
-Enable | With MaintenanceMode: enter maintenance. Without it: leave maintenance |
-RequireNodeInFile | Refuse a -NodeName that is not in nodes[] of the file: exit code 2, nothing is sent to the cluster. The pipelines always pass it |
-Credential | An administrator of the cluster. Without it the account that runs the script is used |
-OutputPath | Also write the rows as JSON to this file |
-LogPath | Also append the progress lines to this file |
| Action | What it does | Changes the cluster |
|---|---|---|
HealthCheck | Nodes Up, pools and virtual disks healthy, no storage repair job running, cluster networks up, the Arc agent of every node connected. Exit 1 when anything is not healthy | no |
MaintenanceMode -Enable | Checks it is safe (other nodes Up, virtual disks healthy, no repair job) and that the node has both Remove-MocPhysicalNode and New-MocPhysicalNode (if not, it stops before anything is drained), then Suspend-ClusterNode -Drain -Wait, then Remove-MocPhysicalNode | yes |
MaintenanceMode | Checks that the node has New-MocPhysicalNode (if not, nothing is resumed), then Resume-ClusterNode -Failback Immediate, New-MocPhysicalNode, then a read of the storage repair jobs | yes |
LiveMigrate | Moves the clustered virtual machines of the node, live, each to the other node with the fewest virtual machines (Move-ClusterVirtualMachineRole -MigrationType Live) | yes |
Rebalance | Spreads the cluster shared volumes evenly over the nodes that are Up (Move-ClusterSharedVolume): one volume at a time moves from the node that owns the most to the node that owns the fewest, until no two nodes differ by more than one volume. For example 2, 2 and 0 volumes become 1, 2 and 1. A balanced cluster moves nothing; a failed move stops the run (exit 1) | yes |
Capacity | Pool size and used space, memory and virtual processors assigned per node | no |
Alerts | Health Service faults, storage health actions and the Critical, Error and Warning events of the failover clustering log of the last 24 hours. Exit 1 when there are any | no |
Run
./Invoke-ClusterOperations.ps1 -InfrastructureFile ./infrastructure.yml -Action HealthCheck -OutputPath ./health.json
./Invoke-ClusterOperations.ps1 -InfrastructureFile ./infrastructure.yml -Action MaintenanceMode -NodeName node01 -Enable -WhatIf
./Invoke-ClusterOperations.ps1 -InfrastructureFile ./infrastructure.yml -Action MaintenanceMode -NodeName node01 -Enable
Exit codes: 0 the action finished (or is planned) and HealthCheck or Alerts found nothing, 1 a step failed or HealthCheck or Alerts found something, 2 cluster.name is missing, the file cannot be read, an action that needs -NodeName has none, or -RequireNodeInFile is given and the node is not in nodes[] (nothing is sent).
To reboot a node in one run, with the wait for the storage repair, use Task 03.
Validation
-
HealthCheckends with exit code 0 and every row isHealthy - After
MaintenanceMode -Enable:Get-ClusterNodeshows the nodePausedandGet-MocPhysicalNodeno longer lists it - After
MaintenanceMode(leave): the node isUp, listed again, andGet-StorageJobshows nothing before the next node is touched
Troubleshooting
| Issue | Cause | Resolution |
|---|---|---|
Not safe: ... on MaintenanceMode -Enable | Another node is not Up, a virtual disk is not healthy or a repair job is running | Wait for the repair to finish (HealthCheck shows it), then run again. Nothing was changed |
The node x does not have Remove-MocPhysicalNode, New-MocPhysicalNode | The node runs a version before 2311.2 | Update first. The run stopped before the drain (or the resume); nothing was changed |
The cluster has no node 'x' | The name is not a node of the cluster | Check -NodeName against nodes[].name |
LiveMigrate fails for one virtual machine | The virtual machine is not clustered, or the target lacks the memory | The other virtual machines are still moved; the exit code is 1. Move that one by hand |
Alerts returns an event of the last 24 hours that is already fixed | The log keeps the event | Read the time of the row; the exit code only clears when no event is left in the window |
Navigation
| Previous | Up | Next |
|---|---|---|
| Task 03: Node Maintenance | Part 8 Index | Implementation Guide |
Version Control
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0.0 | 2026-11-15 | Azure Local Cloud | Initial release |
| 1.0.1 | 2026-11-16 | Azure Local Cloud | MaintenanceMode checks the two Moc cmdlets before it drains or resumes; the Rebalance rule |