Skip to main content

Task 03: Node Maintenance

Implementation Day 2

DOCUMENT CATEGORY: Implementation Runbook
SCOPE: One machine of a running Azure Local cluster (version 2311.2 or later)
PURPOSE: Suspend, drain, reboot and resume a machine, and wait for the storage to repair before another machine is touched
MASTER REFERENCE: Suspend and resume Azure Local machines for maintenance

Status: Active

Do not use this task to install updates: Learn lists manual installs as unsupported, and the update service already drains and reboots the machines in order (Task 02). Use it for a planned reboot or to replace a component.

One machine at a time. The storage of the cluster is shared by all machines, so taking one out leaves the data on the others; while the machine is away, new writes are tracked and, when it returns, resynchronised. Until that repair has finished, taking a second machine out is not safe.


Values from infrastructure.yml​

The values for this task come from infrastructure.yml (schema 5). Every key, with its type, default and example, is in the variables reference.

KeyWhat the script does with it
nodes[].name, nodes[].management.ip_addressFind the machine named by -TargetNode (a name, a fully qualified name or the management address) and its address. A machine that is not in the file is used as it is, with a warning (with -RequireNodeInFile the run stops with exit code 2)
identity.active_directory.domain_fqdnThe fully qualified name name.domain is accepted as -TargetNode
identity.local_admin.username, .passwordThe local administrator the script connects as (the password is a secret reference, read by the toolkit's secret resolver)
identity.active_directory.lcm_user.username, .password, identity.active_directory.netbios_nameThe fallback account when the local administrator is denied. -Credential replaces both

Execution Options​

PowerShell on a cluster machine​

When to use: One machine, by hand (Learn: Suspend and resume Azure Local machines for maintenance; Failover cluster maintenance procedures)

  1. Verify it is safe: every virtual disk is Healthy and OK, no repair job runs, every other machine is Up.
  2. Suspend and drain, then remove the machine from the virtual machine configuration so no new virtual machine lands on it.
  3. Reboot, wait for it, resume it, add it back, and wait for the repair jobs.
Get-VirtualDisk | Format-Table FriendlyName, OperationalStatus, HealthStatus
Get-StorageJob
Suspend-ClusterNode -Name "<MachineName>" -Drain -Wait
Remove-MocPhysicalNode -NodeName "<MachineName>"
Restart-Computer -Force # on the machine
Resume-ClusterNode -Name "<MachineName>" -Failback Immediate
New-MocPhysicalNode -NodeName "<MachineName>"
Get-StorageJob # no output: the repair is done
Get-StoragePool -FriendlyName "<StoragePoolName>"

Validation​

  • Get-ClusterNode shows every machine Up
  • Get-MocPhysicalNode lists the machine
  • Get-StorageJob returns nothing and Get-VirtualDisk shows Healthy and OK for every disk
  • Virtual machines are running on the machine again (with -FailbackMode Immediate)

Troubleshooting​

IssueCauseResolution
Not safe to take 'node' outAnother machine is down, a virtual disk is not healthy, or a repair job is runningWait for the repair to finish (Get-StorageJob), fix the other machine, run again. Nothing was changed
The machine does not have Remove-MocPhysicalNodeThe machine runs a version before 2311.2Update first. The script stops before it changes anything
-SkipDrain needs a node that is already Paused-SkipDrain was given for a machine that is UpRun without -SkipDrain (the script drains it), or suspend it by hand first. Nothing was changed
Restart-Computer failedThe restart command failed with an unexpected errorThe machine stays Paused and nothing was resumed. Check the machine, then run again
There is no proof that node ... rebootedThe last boot time did not change, or it could not be read and the machine was never seen offlineThe machine stays Paused. Check it, reboot it, then run with -ResumeOnly
The machine did not come backHardware or boot problemUse the BMC console. The node stays suspended; resume it with -ResumeOnly once it is back
A repair job stays Suspended at 0 percentThe storage orchestration is waitingDo not reboot again. Check Get-StorageJob and Get-VirtualDisk; the exit code stays 1 until it is repaired

PreviousUpNext
Task 02: Start the Solution UpdatePart 8 IndexTask 04: Cluster Operations

Version Control​

VersionDateAuthorChanges
1.0.02026-11-15Azure Local CloudInitial release. Adds the safety checks, Remove-MocPhysicalNode and New-MocPhysicalNode, the storage repair wait and the exit codes
1.0.12026-11-16Azure Local Cloud-SkipDrain needs a Paused machine; the resume needs proof of the reboot; a failed restart is exit code 1; what exit code 0 means with -SkipResume and -StorageRepairTimeoutMinutes 0; the Resumed field