Task 03: Node Maintenance
DOCUMENT CATEGORY: Implementation Runbook
SCOPE: One machine of a running Azure Local cluster (version 2311.2 or later)
PURPOSE: Suspend, drain, reboot and resume a machine, and wait for the storage to repair before another machine is touched
MASTER REFERENCE: Suspend and resume Azure Local machines for maintenance
Status: Active
Do not use this task to install updates: Learn lists manual installs as unsupported, and the update service already drains and reboots the machines in order (Task 02). Use it for a planned reboot or to replace a component.
One machine at a time. The storage of the cluster is shared by all machines, so taking one out leaves the data on the others; while the machine is away, new writes are tracked and, when it returns, resynchronised. Until that repair has finished, taking a second machine out is not safe.
Values from infrastructure.yml
The values for this task come from infrastructure.yml (schema 5). Every key, with its type, default and example, is in the variables reference.
| Key | What the script does with it |
|---|---|
nodes[].name, nodes[].management.ip_address | Find the machine named by -TargetNode (a name, a fully qualified name or the management address) and its address. A machine that is not in the file is used as it is, with a warning (with -RequireNodeInFile the run stops with exit code 2) |
identity.active_directory.domain_fqdn | The fully qualified name name.domain is accepted as -TargetNode |
identity.local_admin.username, .password | The local administrator the script connects as (the password is a secret reference, read by the toolkit's secret resolver) |
identity.active_directory.lcm_user.username, .password, identity.active_directory.netbios_name | The fallback account when the local administrator is denied. -Credential replaces both |
Execution Options
- PowerShell on a cluster machine
- PowerShell
PowerShell on a cluster machine
When to use: One machine, by hand (Learn: Suspend and resume Azure Local machines for maintenance; Failover cluster maintenance procedures)
- Verify it is safe: every virtual disk is
HealthyandOK, no repair job runs, every other machine isUp. - Suspend and drain, then remove the machine from the virtual machine configuration so no new virtual machine lands on it.
- Reboot, wait for it, resume it, add it back, and wait for the repair jobs.
Get-VirtualDisk | Format-Table FriendlyName, OperationalStatus, HealthStatus
Get-StorageJob
Suspend-ClusterNode -Name "<MachineName>" -Drain -Wait
Remove-MocPhysicalNode -NodeName "<MachineName>"
Restart-Computer -Force # on the machine
Resume-ClusterNode -Name "<MachineName>" -Failback Immediate
New-MocPhysicalNode -NodeName "<MachineName>"
Get-StorageJob # no output: the repair is done
Get-StoragePool -FriendlyName "<StoragePoolName>"
PowerShell
When to use: The same procedure with the checks, the waits and the exit codes
| Script | What it does |
|---|---|
scripts/deploy/08-day2-operations/task-03-node-maintenance/powershell/Invoke-NodeMaintenanceReboot.ps1 | Checks, drains, removes from the VM configuration, reboots, waits, resumes, adds back, waits for the storage repair. -WhatIf sends nothing to any machine |
| Parameter | Meaning |
|---|---|
-InfrastructureFile | The infrastructure.yml (schema 5). Required |
-TargetNode | The machine: name, fully qualified name or management address. Required |
-RequireNodeInFile | Refuse a -TargetNode that is not in nodes[] of the file: exit code 2, before any credential is read and before anything is sent. The pipelines always pass it |
-Credential | Use this account instead of the ones in the file |
-SkipDrain | Reboot without the drain (no Suspend-ClusterNode, no Remove-MocPhysicalNode). The machine must already be Paused: the script reads its state first and stops with exit code 1, with nothing changed, when it is not. The other safety checks still run |
-SkipResume | Leave the machine in maintenance after the reboot. Run again with -ResumeOnly |
-ResumeOnly | Only resume a machine that is in maintenance, add it back and wait for the storage |
-FailbackMode | Immediate (default), NoFailback or Policy |
-TimeoutSeconds, -PingIntervalSeconds | How long, and how often, to wait for the machine to come back |
-StorageRepairTimeoutMinutes, -StoragePollSeconds | How long (0 to 1440, default 180; 0 does not wait) and how often to wait for the repair jobs |
-LogPath | The log file. Default logs\node-maintenance-reboot\<date>_<time>_<node>.log |
Run
./Invoke-NodeMaintenanceReboot.ps1 -InfrastructureFile ./infrastructure.yml -TargetNode node01 -WhatIf
./Invoke-NodeMaintenanceReboot.ps1 -InfrastructureFile ./infrastructure.yml -TargetNode node01
The reboot must be proven before the machine is resumed. The script reads the last boot time (Win32_OperatingSystem.LastBootUpTime) before and after the restart. The machine is resumed only when the boot time is newer. When a boot time cannot be read, the machine must have been seen offline instead. Without that proof the run ends with exit code 1 and the machine stays Paused: check it, reboot it, then run with -ResumeOnly. If Restart-Computer fails with an error other than the dropped connection of a reboot, the run also ends with exit code 1 and the machine stays Paused.
Run it for the next machine only after the first has ended with exit code 0 and its result shows Resumed true and StorageRepair Repaired: the machine is Up, in the virtual machine configuration again, and the storage is repaired.
| Exit code | Meaning |
|---|---|
0 | Done, and the storage is healthy (or planned under -WhatIf). With an opt-out it means less: see below |
1 | It was not safe (nothing was changed), -SkipDrain was given for a machine that is not Paused (nothing was changed), the machine was not Paused after the drain, the restart failed, there is no proof that the machine rebooted, it did not come back, a step failed, or the storage had not repaired when the wait ran out |
2 | The file is missing or not usable, or no credential could be built |
What exit code 0 means with the two opt-outs:
| Run with | Exit code 0 means |
|---|---|
-SkipResume | The machine rebooted and came back, and it is still Paused. It was not resumed, it is not in the virtual machine configuration again, and the storage was not checked (Resumed false, StorageRepair NotWaited). Run -ResumeOnly next |
-StorageRepairTimeoutMinutes 0 | The machine rebooted, was resumed and is Up, but the storage repair was not waited for (StorageRepair NotWaited). Do not take another machine offline until Get-StorageJob shows no job and every virtual disk is Healthy |
The result object has Node, NodeIp, State, DrainStatus, Resumed (true when the run resumed the machine), StorageRepair (Repaired or NotWaited) and Duration (seconds).
Validation
-
Get-ClusterNodeshows every machineUp -
Get-MocPhysicalNodelists the machine -
Get-StorageJobreturns nothing andGet-VirtualDiskshowsHealthyandOKfor every disk - Virtual machines are running on the machine again (with
-FailbackMode Immediate)
Troubleshooting
| Issue | Cause | Resolution |
|---|---|---|
Not safe to take 'node' out | Another machine is down, a virtual disk is not healthy, or a repair job is running | Wait for the repair to finish (Get-StorageJob), fix the other machine, run again. Nothing was changed |
The machine does not have Remove-MocPhysicalNode | The machine runs a version before 2311.2 | Update first. The script stops before it changes anything |
-SkipDrain needs a node that is already Paused | -SkipDrain was given for a machine that is Up | Run without -SkipDrain (the script drains it), or suspend it by hand first. Nothing was changed |
Restart-Computer failed | The restart command failed with an unexpected error | The machine stays Paused and nothing was resumed. Check the machine, then run again |
There is no proof that node ... rebooted | The last boot time did not change, or it could not be read and the machine was never seen offline | The machine stays Paused. Check it, reboot it, then run with -ResumeOnly |
| The machine did not come back | Hardware or boot problem | Use the BMC console. The node stays suspended; resume it with -ResumeOnly once it is back |
A repair job stays Suspended at 0 percent | The storage orchestration is waiting | Do not reboot again. Check Get-StorageJob and Get-VirtualDisk; the exit code stays 1 until it is repaired |
Navigation
| Previous | Up | Next |
|---|---|---|
| Task 02: Start the Solution Update | Part 8 Index | Task 04: Cluster Operations |
Version Control
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0.0 | 2026-11-15 | Azure Local Cloud | Initial release. Adds the safety checks, Remove-MocPhysicalNode and New-MocPhysicalNode, the storage repair wait and the exit codes |
| 1.0.1 | 2026-11-16 | Azure Local Cloud | -SkipDrain needs a Paused machine; the resume needs proof of the reboot; a failed restart is exit code 1; what exit code 0 means with -SkipResume and -StorageRepairTimeoutMinutes 0; the Resumed field |