Using PowerShell Azure Functions to monitor Storage Queues

I’ve been doing some work recently using Azure Storage Queues with Azure Functions to ingest data from a third party. The solution consists of:

• an HTTP triggered PowerShell Azure function the third party sends data in via HTTP
• a Storage Account containing a queue that inbound data is loaded onto
• a queue triggered PowerShell Azure Function that processes the inbound messages.

If the queue processing function hasn’t successfully processed the message after 1 minute, it goes back on the queue to be retried. If it fails after 10 attempts, then it disappears from the queue and is moved to another queue (similary named by with the suffix “-poison”) to be investigated (this is all configured in the host.json file in the Function App).

As part of my monitoring setup, I wanted to be able to monitor the number of messages in both the ingestion queue and the ingestion-poison queue.

But…

Looking at the metrics available, there is one that gives you the number of messages across all the queues in the Storage Account, but no metrics for the number of messages in the individual queues.

I’ve seen posts where people have written functions in C# to query the number of messages in queues. I wanted to do something in PowerShell, as it’s a language I’m pretty comfortable with. I also wanted to record the metrics for putting on a dashboard.

Getting the number of messages an a queue

You can queue the number of messages using the Az PowerShell modules, as below:

1
2
3
4
5
6
7
8
9

$storageAccountName = "stautodevuks01"
$resourceGroupName = "rg-testqueues"
$queue = "ingestion"

$storageaccount = Get-AzStorageAccount -StorageAccountName stautodevuks01 -ResourceGroupName $resourceGroupName

$storageContext = New-AzStorageContext -StorageAccountName $storageAccount.StorageAccountName

$messageCount = (Get-AzStorageQueue -Context $storageContext | where-object{$_.name -eq $queue}).ApproximateMessageCount

But, I didn’t want the added dependencies, so hadn’t been using any of the Az module cmdlets in the other functions, so I did this using the Azure Queue Storage API instead.

You can get the number of messages in a queue in the response to a Get Metadata request as describes in the Microsoft documentation.

Authentication to the Queue Storage API is a little tricky - needing to first calculate a signature containing the request info, hashing it with the access key then converting to base64. Although you don’t need to include many values in the signature, those you omit still need a newline character.

The following PowerShell queries the Queue Storage API for the number of messages in a queue:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

$storageAccountName = "stautodevuks01"
$accessKey = "Kv0eVZEdATTBjP5bNhjnzlW7h0F1TngT7pBNLilKblfJKfjyKTUEbrL0SVYcSjHaUf4Sy4u6bDJt+AStuYzTKA=="
$queue = "ingestion"

$apiVersion = "2021-12-02"
$queue_url = "https://$storageAccountName.queue.core.windows.net/$queue" + "?comp=metadata"
$apiCallTimestamp = [DateTime]::UtcNow.ToString('r')

# Generate signature string based on the request we are about to make to the Azure Storage Queue API
$signatureString_canonicalizedHeaders = "x-ms-date:$apiCallTimestamp`nx-ms-version:$apiVersion`n"
$signatureString_canonicalizedResource = "/$storageAccountName/$queue" + "`ncomp:metadata"
$stringToSign = "GET" + "`n`n`n`n`n`n`n`n`n`n`n`n" + `
    $signatureString_canonicalizedHeaders + `
    $signatureString_canonicalizedResource

# Hash the signature using the Storage Account Key and convert to Base64 to use in the Authorization header
$hmac = New-Object System.Security.Cryptography.HMACSHA256
$hmac.key = [Convert]::FromBase64String($accessKey)
$signatureUTF8 = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign))
$signatureBase64 = [Convert]::ToBase64String($signatureUTF8)

# Build the headers for the request
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("x-ms-date", $apiCallTimestamp)             
$headers.Add("x-ms-version", $apiVersion)      
$headers.Add("Authorization","SharedKey $($storageAccountName):$signatureBase64")

# Make the request to the Azure Storage Queue API and return the message count
Invoke-RestMethod -Method "GET" -Uri $queue_url -Headers $headers -ResponseHeadersVariable 'ResponseHeaders' | Out-Null
$messageCount = $responseHeaders['x-ms-approximate-messages-count']

# Write an information log with the current time.
Write-Host("Checked message count for '$queue' - $messageCount messages")

Creating custom metrics

Ryland DeGregory wrote a great post on Using Azure Application Insights SDK with PowerShell which describes how to log events and exceptions into App Insights from PowerShell, despite it not being officially supported.

The short version of how to do this:

1. Download the .NET SDK for App Insights from NuGet
2. Extract the contents - you can rename it to a .zip to open it in Windows Explorer
3. Copy the Microsoft.ApplicationInsights.dll file from the “net452” folder into the folder for the function
4. Add the following code to the PowerShell function to setup the connection to App Insights

1
2
3
4
5
6
7
8

# Load .dll assembly into PowerShell session
[Reflection.Assembly]::LoadFile("$PSScriptRoot\Microsoft.ApplicationInsights.dll")

# Instanciate a new TelemetryClient
$TelemetryClient = [Microsoft.ApplicationInsights.TelemetryClient]::new()

# Set the Application Insights Instrumentation Key
$TelemetryClient.InstrumentationKey = '<AppInsights Instrumentation Key>'

Ryland describes how to do publish events and exceptions, but I found that metrics were pretty simple too. You create a dictionary object with the key:value pair that you want to push into App Insights as a metric and then used TelemetryClient.TrackMetric and pass the values in. You have to do a TelemtryClient.Flush() to actually send the data. I think the idea is that, in a normal application, you’d probably batch up your logs and metrics and “flush” to send them all in on a single call to App Insights:

1
2
3
4
5

$metric = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$metric.add("name", "queue-$queue MessageCount")
$metric.add("value", $messageCount)
$TelemetryClient.TrackMetric($metric)
$TelemetryClient.Flush()

Putting it all together

Given that most of it is detailed above, I’ve not posted the function in full here, but have committed the full version of the script to Github:

https://github.com/davelee212/azfunc-posh-queue-monitor

Getting it running:

1. Create an App Insights instance
2. Deploy the function to a PowerShell Function App
3. Put the required variables into the Function App Configuration (the Storage Account Name, Access Key, App Insights instrumentation key and a comma separated list of the queues to monitor)
4. Save the Configuration and restart the Function App

The function then runs every 5 minutes to publish the message counts on each queue into App Insights.

And we can build charts against the metrics and setup alerts when the message count gets higher than expected so we can investigate possible performance issues. It seems to do the job nicely and runs right alongside the other functions in the same Function App.