Part 8: Day 2 Operations
DOCUMENT CATEGORY: Implementation Runbook
SCOPE: Azure Local clusters that are deployed and handed over
PURPOSE: Keep the cluster current and healthy: the monthly update, node maintenance and the day to day cluster operations
MASTER REFERENCE: Update Azure Local by using PowerShell
Status: Active
Every task of this part has a script under scripts/deploy/08-day2-operations that reads infrastructure.yml, a page here and a test. Nothing in this part runs by itself: an operator, a pipeline or a scheduled task starts each script.
Tasks
| Task | What it does | Script folder | Azure CLI script |
|---|---|---|---|
| Task 01: Check for Updates | Reads what the update service has discovered and what Azure shows. Read only. Run it after the deployment and on a schedule | scripts/deploy/08-day2-operations/task-01-check-for-updates | yes |
| Task 02: Start the Solution Update | Starts (or only prepares) the chosen solution update, after the deployment or for the monthly update | scripts/deploy/08-day2-operations/task-02-start-solution-update | no |
| Task 03: Node Maintenance | Checks it is safe, drains, reboots and resumes one node, then waits for the storage to repair | scripts/deploy/08-day2-operations/task-03-node-maintenance | no |
| Task 04: Cluster Operations | Health check, maintenance mode, live migration, volume ownership, capacity and alerts | scripts/deploy/08-day2-operations/task-04-cluster-operations | no |
Each task also has a pipeline on GitHub Actions, GitLab CI and Azure DevOps: the check for updates on a schedule, and every change after a read-only plan, behind one approval environment. See Run Day 2 Operations in a Pipeline.
The monthly update, in order
An Azure Local release is supported for six months, so a cluster takes an update about once a month. The steps are always the same (Update management and SBE has the background):
- Check. Task 01. The verdict says what is offered:
UpdateAvailable,AdditionalContentRequired,UpdateRunning,UpToDateorAttentionNeeded. - Import the SBE files, only when an update is in the state
AdditionalContentRequired. Phase 06, Task 11: SBE update import lists the files on the share and prints the import commands. Then check again (step 1) until the update isReady. - Prepare (recommended). Task 02 with
-Mode OnDemand -PrepareOnlydownloads the update and runs the readiness checks before the maintenance window. The update becomesReadyToInstall. - Start. Task 02 with
-Mode OnDemand, inside the maintenance window. The script returns at once; the update takes hours and the machines restart. - Follow. Run Task 01 again, or watch Update Manager in the Azure portal. Sonar watches the progress as well.
- Verify. The update is
InstalledandGet-SolutionUpdateEnvironmentshows the newCurrentVersion. Run the checks of Part 7 again.
The first update after a deployment is the same path. Set operations.updates.install_after_deployment: true to let Task 02 start it in the default mode AfterDeployment.
Check after the deployment, and on a schedule
A freshly deployed cluster does not show its first update on its own until later: the update service finds updates by itself, and the update agent checks once a day (Microsoft Learn: Use Azure Update Manager to update Azure Local). So the toolkit always checks:
- Right after the deployment, once:
Get-AvailableSolutionUpdate.ps1 -WaitMinutes 120waits for the first update to be offered. - On a schedule, for example daily:
Get-AvailableSolutionUpdate.azcli.ps1 -OutputPath <file>needs no access to a node, so a pipeline agent or an administrator's workstation can run it. Exit code1means the cluster needs attention (a failed update, a health state ofFailure,ErrororWarning, or an update service that has not checked for 48 hours): use it as the alert. The JSON file holds the verdict and every update for a monitor.
Both scripts only read. Neither installs an update.
Prerequisites
- The cluster is deployed (Part 4) and registered in Azure
-
infrastructure.ymlis the file the deployment used (cluster.name;azure.*for the Azure CLI scripts) - The account that runs the PowerShell scripts is the deployment user (Learn: sign in with the deployment user account), or
-Credentialnames it - For the Azure CLI script:
az login --tenant <azure.tenant_id>was run, with at least read access to the cluster resource
Navigation
| Previous | Up | Next |
|---|---|---|
| Part 7: Validation & Handover | Implementation Guide | Task 01: Check for Updates |
Document maintained by Azure Local Cloud Team. Part of the Azure Local Implementation Guide.