Skip to content

VMFleet Monitoring ​

Tool: VMFleetCategory: Tool Guide

The monitoring subsystem collects real-time storage, network, and compute metrics during VMFleet test execution. It uses a hybrid architecture: local PowerShell/PerfMon collection (always active) with optional push to Azure Monitor/Log Analytics.

Monitoring Architecture ​

text
Cluster Nodes (PerfMon counters)
        │
        ▼
PowerShell Get-Counter (remote sessions)
        │
        ├──► Local CSV/JSON files (results/{run-id}/metrics/)
        │
        └──► Azure Monitor / Log Analytics (optional)
                    │
                    ▼
              Azure Portal dashboards, alerts

Live Console Monitoring ​

Use the built-in watcher during active tests:

powershell
.\src\solutions\vmfleet\scripts\Watch-VMFleetMonitor.ps1 `
    -ClusterConfigPath "config/clusters/my-cluster.yml" `
    -RefreshIntervalSeconds 5 `
    -DurationMinutes 30

The watcher prints rolling VM count, read/write IOPS, read/write latency, and throughput.

Metric Categories ​

Storage Metrics ​

Collected by Collect-StorageMetrics.ps1:

CounterDescription
CSVFS: Read IOPSCluster Shared Volume read operations per second
CSVFS: Write IOPSCluster Shared Volume write operations per second
CSVFS: Read Throughput (MB/s)Aggregate read bandwidth across CSVs
CSVFS: Write Throughput (MB/s)Aggregate write bandwidth across CSVs
CSVFS: Read Latency (ms)Average read latency per I/O operation
CSVFS: Write Latency (ms)Average write latency per I/O operation
S2D Cache Hit RatioPercentage of I/O served from cache tier
Physical Disk Queue DepthOutstanding I/O requests per physical disk
Storage Pool HealthDegraded/healthy status of storage pools

Network Metrics ​

Collected by Collect-NetworkMetrics.ps1:

CounterDescription
RDMA Activity: Bytes Sent/secRDMA network adapter transmit throughput
RDMA Activity: Bytes Received/secRDMA network adapter receive throughput
SMB Direct: Bytes Sent/secSMB Direct (RDMA) data transfer rate
Network Adapter: Throughput (Gbps)Total NIC throughput per adapter
Network Adapter: Packets DroppedDropped packets indicating congestion or errors
TCP: Retransmits/secTCP retransmission rate (indicates network issues)

Compute Metrics ​

Collected by Collect-ComputeMetrics.ps1:

CounterDescription
Processor: % Total CPUHost CPU utilization across all logical processors
Hyper-V Hypervisor Logical Processor: % Total Run TimeHypervisor-level CPU consumption
Guest vCPU UtilizationPer-VM CPU usage (via VMFleet Watch-FleetCPU)
Memory: Available MBytesFree physical memory on each host
Memory: % Committed Bytes In UseMemory pressure indicator
Hyper-V Dynamic Memory: Current PressureDynamic memory pressure per VM

Local Metric Collection ​

Local collection is always active during test runs:

powershell
# Collect all metric categories with 5-second intervals
.\tools\vmfleet\monitoring\Collect-StorageMetrics.ps1 `
    -ClusterConfig "config/clusters/my-cluster.yml" `
    -SampleIntervalSeconds 5 `
    -OutputPath "results/run-001/metrics/storage/"

.\tools\vmfleet\monitoring\Collect-NetworkMetrics.ps1 `
    -ClusterConfig "config/clusters/my-cluster.yml" `
    -SampleIntervalSeconds 5 `
    -OutputPath "results/run-001/metrics/network/"

.\tools\vmfleet\monitoring\Collect-ComputeMetrics.ps1 `
    -ClusterConfig "config/clusters/my-cluster.yml" `
    -SampleIntervalSeconds 5 `
    -OutputPath "results/run-001/metrics/compute/"

Output format is JSON-lines for machine parsing:

json
{"timestamp":"2026-02-13T10:05:00Z","node":"iic-azl-n01","counter":"CSVFS_ReadIOPS","value":45230}
{"timestamp":"2026-02-13T10:05:00Z","node":"iic-azl-n01","counter":"CSVFS_WriteIOPS","value":12840}

Azure Monitor Integration ​

Optionally push collected metrics to Azure Monitor for centralized dashboards and alerting:

powershell
# Workspace key from the LOG_ANALYTICS_KEY environment variable or the log_analytics_key Key Vault mapping
.\tools\vmfleet\monitoring\Push-MetricsToAzureMonitor.ps1 `
    -MetricsPath "results/run-001/metrics/" `
    -WorkspaceId "00000000-0000-0000-0000-000000000000"

The script takes no credential-source switch. The workspace ID comes from -WorkspaceId, otherwise from log_analytics_workspace_id in tools/vmfleet/config/vmfleet.json; without one it logs a warning and pushes nothing. The workspace key comes from -WorkspaceKey, then the LOG_ANALYTICS_KEY environment variable, then the log_analytics_key mapping in config/credentials/keyvault-config.yml (sign in to Azure first). When none has a value the script stops with exit code 2; nothing prompts. See Credential Management.

Each pushed metric contains these core fields:

  • timestamp
  • node
  • counter_name
  • value
  • run_id
  • profile_name

NOTE

Azure Monitor integration requires the monitoring.bicep infrastructure template to be deployed. See tools/vmfleet/infrastructure/bicep/monitoring.bicep.

Alert Rules ​

Predefined alert thresholds in monitoring/alerts/alert-rules.yml:

yaml
alerts:
  - name: high_storage_latency
    metric: csvfs_write_latency_ms
    threshold: 50
    severity: warning
    description: "Write latency exceeds 50ms — possible storage bottleneck"

  - name: critical_cpu
    metric: host_cpu_percent
    threshold: 95
    severity: critical
    description: "Host CPU exceeds 95% — test may be compute-bound"

Real-Time Dashboard ​

During test execution, use the monitoring dashboard wrapper:

powershell
# Launch real-time monitoring (combines VMFleet Watch-FleetCluster with custom metrics)
.\tools\vmfleet\monitoring\Export-MetricsDashboard.ps1 `
    -InputPath "results/run-001/metrics/" `
    -OutputPath "reports/run-001/" `
    -Title "VMFleet Run 001"

Counter Reference ​

CategoryCounter namePurpose
StorageCSVFS_ReadIOPSCSVFS read operations per second
StorageCSVFS_WriteIOPSCSVFS write operations per second
StorageCSVFS_ReadMBpsRead throughput
StorageCSVFS_WriteMBpsWrite throughput
StorageCSVFS_ReadLatencyMsRead latency
StorageCSVFS_WriteLatencyMsWrite latency
ComputeHostCpuPercentHost CPU saturation
ComputeHostAvailableMemoryMBAvailable memory headroom
ComputeHyperVLogicalProcessorRunTimeHypervisor CPU pressure

KQL Examples ​

kusto
VMFleetMetrics_CL
| where CounterName_s in ("CSVFS_ReadIOPS", "CSVFS_WriteIOPS")
| summarize AvgValue=avg(Value_d) by bin(TimeGenerated, 1m), CounterName_s
| render timechart
kusto
VMFleetMetrics_CL
| where CounterName_s in ("CSVFS_ReadLatencyMs", "CSVFS_WriteLatencyMs")
| summarize P95=percentile(Value_d, 95) by bin(TimeGenerated, 1m), CounterName_s
| render timechart

Released under the MIT License.