Skip to main content

Monitor the Run with Sonar

Runbook Azure Preview

DOCUMENT CATEGORY: Runbook SCOPE: Watching an azurelocal-ztp run with azurelocal-sonar PURPOSE: Start Sonar for a ZTP run, read what it shows, and keep it running into cluster deployment MASTER REFERENCE: azurelocal-sonar documentation

Status: Active


Overview​

azurelocal-sonar is a local web page that follows an Azure Local provisioning and deployment live. For a ZTP run it shows, on one page:

  • every ZTP stage as it starts and ends, from the virtual USB boot to the voucher;
  • each machine's provisioning in Azure: its lifecycle, its stages and the ProvisionOs job steps;
  • each machine's Prepare for Validate stage: drivers installed, restart, ready for Validate;
  • once you deploy the cluster, Validate and Deploy, with the seed node's logs.

Sonar only reads. It never resubmits a job, never changes Azure, and never touches the machines or the event file. The ZTP run checks every stage itself and stops when a check fails, so Sonar is for watching, not for control. You don't need it for the run to succeed.

LinkWhat it is
Sonar documentationThe Sonar site
Getting startedInstall, sign-in, and which account reads the node logs
Following a ZTP runWhat Sonar reads during ZTP, how it decides running, ready or failed, and every config key
ZTP: Monitoring a runWhat each ZTP stage checks, and the event file format
azurelocal-sonar repositorySource and releases

What Sonar reads​

SourceWhat it givesHow
The ZTP event file ztp-events.jsonlOne line per stage start, end and failure, the boot and voucher events, and the Prepare eventsReads the file on the management host every 2 seconds, without locking it
AzureThe edge machines, their ProvisionOs job and lifecycle stages, and each Arc machineRead-only ARM GET requests under your Az PowerShell sign-in, every 20 seconds by default
The seed nodeValidate and Deploy logsWinRM, once the cluster deployment starts

Prerequisites​

RequirementDetails
WhereThe same management host as the ZTP run. Sonar reads the event file from the local disk.
PowerShell7.4 or later
Az sign-inAz.Accounts, signed in with Connect-AzAccount
Azure roleRead on the resource group of the edge machines and on the managed resource group that holds their Arc machines. Reader on the subscription covers both.
ZTP inputsThe ZTP plan or run has written config\simplified-provisioning\nodes.json in the ZTP clone. The plan run writes it, so you can start Sonar before -Execute.

Step-by-Step​

Step 1: Get Sonar​

Clone the Sonar repository next to the ZTP clone.

git clone https://github.com/AzureLocal/azurelocal-sonar.git D:\git\azurelocal-sonar

Step 2: Start Sonar from the ZTP clone​

Open a second PowerShell 7 window, so the ZTP run keeps its own. Change to the root of the ZTP clone and sign in.

Set-Location D:\git\azurelocal-ztp
Connect-AzAccount -Tenant <tenant-id>

Run Start-ZtpMonitor.ps1 without -Execute first. It prints the Sonar config it would write and the command, and changes nothing.

.\scripts\simplified-provisioning\Start-ZtpMonitor.ps1 `
-SonarPath D:\git\azurelocal-sonar `
-SubscriptionId <subscription-id> `
-ResourceGroupName <resource-group> `
-NodesFile .\config\simplified-provisioning\nodes.json `
-EventLogPath .\_media\ztp-state\ztp-events.jsonl `
-ClusterName <cluster-name>

Use the values from your environment file: ztp.subscription_id and ztp.resource_group. Pass the real name of the cluster you will deploy as -ClusterName, so Sonar goes on to follow Validate and Deploy. Add -IdentityModel LocalIdentity if the cluster won't use Active Directory; the default is ActiveDirectory.

When the output looks right, run it again with -Execute. It writes ztp-sonar.config.json in the current folder and starts Sonar.

Sonar asks once for an account to read the seed node's logs. During provisioning there are no logs to read on the nodes: cancel, and Sonar uses your Windows identity. See Which account for Validate and Deploy.

ParameterUse
-NodeNameFollow only some machines
-PortLocal port of the page, default 8642
-NoBrowserDon't open the browser
-Record <file>Also record the run, for replay later. Recording is off unless you pass it.

Step 3: Open the page​

Sonar opens http://localhost:8642/ in your browser, or the -Port you chose. The page updates by itself.

Step 4: Read what you see​

Part of the pageWhat it shows during the run
HeaderThe cluster name, the phase chip, the overall status such as "1 of 2 machines ready", elapsed time, last poll, and one chip per machine
Provisioning panelOne item per machine: lifecycle, progress bar, Arc machine, extensions, and the lifecycle message. Expand it for the lifecycle stages, the ProvisionOs job steps with their timers, and Prepare for Validate
Live logsThe [ztp] lines from the event file, by time and coloured by level. Pick ztp-events in the file filter to see only those.

The run is done when every machine's Prepare stage shows ready for Validate and the phase chip reads Ready for Validate. That matches the run's own last line, <node> is ready for Validate.

Step 5: Leave it running for cluster deployment​

When you deploy the cluster with the name you gave as -ClusterName, Sonar moves on to Validation and then Deployment, and follows the seed node's logs. See Following a ZTP run and Sonar's Getting started.


What's normal​

These were seen on the proven runs, on Dell AX-760 with iDRAC 7.30. They recover by themselves; see Troubleshooting and known issues.

What you seeWhat it means
A machine at AwaitingConnection for a long time after the claimIts registration is late. On the proven runs it took up to about 45 minutes, and recovered on its own.
A machine turns failed at SetupHybridComputeRP with 5901, then running again a minute laterThe first OS install attempt failed, and ZTP resubmitted it once. Sonar shows the failure while the job reports it. Once the new attempt starts, the old error moves to the machine's previous errors and the machine is running again. It didn't happen on every machine.
A machine stays running after Azure reports ProvisionedZTP's Prepare stage is still installing drivers and setting the network. The machine is done when Prepare reports ready for Validate.
A restart in the Prepare stageA driver asked for it. Prepare waits for the machine to come back with a new boot time and carries on.

If Sonar and the run disagree, the run is right: it stops with the reason when a check fails. The Azure portal also shows each machine under Azure Arc > Operations > Provisioning (preview) > Provisioned machines.


Validation​

  • Sonar shows every machine of the run
  • The live logs show [ztp] lines from the event file
  • At the end, every machine shows ready for Validate and the phase chip reads Ready for Validate

← Run from a management host↑ Simplified ProvisioningRun from a pipeline →

References​


VersionDateAuthorChanges
1.02026-11-06Azure Local CloudInitial release