Azure Storage Queue Monitoring

Microsoft Azure Storage queues, which are part of the Azure storage infrastructure, feature a simple REST-based GET/PUT/PEEK interface, providing reliable, persistent messaging within and between services. They are designed for large cloud networks, or hybrid networks, providing a highly reliable queuing service.

Azure Storage Queue is also notable because it allows you to access messages from literally any device in the world with an active Internet connection, all thanks to authenticated calls using HTTP or HTTPS. A queue could potentially contain millions of messages depending on the situation, up to the total capacity limit of the Azure storage account in question. These queues are also commonly used to create a backlog of work to process asynchronously, creating a better and more informed developer environment whenever possible.

Queues are typically addressable using the following URL format:

• https://<storage account>.queue.core.windows.net/<queue>

Prior to the 2017 release of Azure Storage Queue, the maximum time a message was allowed to remain in the queue was seven days. After that point, messages no longer expire - though if you don't specify a particular amount of time for the message to remain, it will still default to that seven day period.

Why use a Telegraf plugin for Azure Storage Queue?

The Microsoft Azure Storage Queue Telegraf plugin collects the sizes of the Microsoft Azure Storage Queues. A single queue message can be up to 64 KB in size, and a queue can contain millions of messages, up to the total capacity limit of a storage account.

In general, Azure Storage Queue brings with it a number of unique benefits for developers everywhere that they simply cannot afford to ignore. It can be used to build flexible applications and to separate functions through decoupled components, for example, thus guaranteeing better durability across large workloads. When designing applications for scale, this allows you to scale each component independently of one another. Azure Storage Queue then gives you asynchronous message queuing for communication between those application components. This is true regardless of whether they are running in the cloud, on a desktop, on-premises or even on mobile devices.

Likewise, Azure Storage Queue is a great way to rightsize your service deployment. Thanks to Azure Storage Queue, applications can absorb unexpected traffic bursts, which itself goes a long way from preventing your servers from being overwhelmed (and potentially taken offline) by a sudden flood of requests. You can also monitor queue length to add as much elasticity as possible to your application, and can use it to deploy or hibernate additional worker nodes based on the real-time demand of your customers.

Azure Storage Queue helps to make applications both scalable and less sensitive to individual component failure, all in the name of guaranteeing the best performance possible for end users. If a single part of your architecture goes down unexpectedly, or if messages are buffered, things are naturally picked up by other processing nodes. This goes a long way towards helping to maintain the integrity of your workload, regardless of what life happens to throw at you.

How to monitor Azure Storage Queue using the Telegraf plugin

Setting up the Microsoft Azure Storage Queue Telegraf plugin simply requires that you configure the Azure Storage Account name and the access key. Once this is completed, you can start sending storage queue metrics into InfluxDB.

To begin setting up the Azure Storage Queue Telegraf plugin on your own environment, simply use the following configuration commands. Just replace all of the default values with the most accurate information that you’re working with given the specifics of your deployment:

# Description
[[inputs.azure_storage_queue]]
  ## Required Azure Storage Account name
  account_name = "mystorageaccount"

  ## Required Azure Storage Account access key
  account_key = "storageaccountaccesskey"

  ## Set to false to disable peeking age of oldest message (executes faster)
  # peek_oldest_message_age = true

Key Azure Storage Queue Metrics to use for monitoring

The Azure Storage Queue metrics that you will get include:

• queue
• size (integer, count)
• oldest_message_age_ns (integer, nanoseconds) Age of message at the head of the queue. Requires peek_oldest_message_age to be configured to true.

Project URL   Documentation