Azure Monitoring Tools Explained – Part 5: Azure Alerts

In the first part of this series, we introduced the confusion and complexity that tends to occur when looking at the long list of monitoring tools available for Azure.

We then provided a list of currently available tools that we will explore further.

• Part 2: Activity Logs
• Part 3: Application Insights
• Part 4: Azure Advisor
• Part 5: Azure Alerts
• Part 6: Azure Diagnostics
• Part 7: Azure Metrics
• Part 8: Azure Monitor
• Part 9: Azure Security Center (ASC)
• Part 10: Network Watcher
• Part 11: Operations Management Suite (OMS)
• Part 12: Service Health
• Part 13: System Center Operation Manager (SCOM)

We’ve already discussed Azure Activity Logs, Application Insights, and Azure Advisor. The next tool on the list is Azure Alerts. 

Azure Alerts

Alerts are a simple concept. Either something did or did not happen, and you want to be notified to this. You can think of common things like:

• Dramatic increases in CPU/RAM, or low disk space
• Objects were created/deleted
• Permissions were changed (i.e. usually granted)

Azure Alerts actually spans across multiple areas. You can create Alerts from Azure Activity Logs, or Azure Metrics, or Operations Management Suite, etc. And you can then react to those through notifications and even automation.

Alerts, as with a lot of the toolsets in this series, falls within the Azure Monitor umbrella, and really, you’ll start to recognize a pattern that Azure Monitor is becoming the central location for all monitoring toolsets in Azure, but we’ll expand on that in the Azure Monitor part of this series.

There is an excellent table (in the Alerts In Different Azure Services article) that lists and explains the different types of Alerts (i.e. Metrics, Activities, Analytics, etc.) and which monitoring tool/service to use. I’ll reproduce this table here in a simplified structure, but all credit goes to Microsoft.

Here is an overview of Alerts in Azure: https://learn.microsoft.com/en-ca/azure/azure-monitor/alerts/alerts-overview#alerts-in-different-azure-services

Real Word Example

So let’s actually put some hands-on with this to get a better idea of how/when to use it.

First, navigate to Azure Monitor, and click on Alerts under the Shared Services section.

You may notice that in the header that not all options are available. This is because some of these mechanisms are specific to certain types of resources (i.e. Metric Alerts against Virtual Machines).

Activity Log Alert

I’m going to start with creating an Alert against a specific activity (i.e. Virtual Machine deallocation, aka shutdown). I want to receive an Alert when a VM is shutdown (assuming that it should not ever be shutdown, like a Domain Controller).

When I click on “+ Add Activity Log Alert” I am presented with a dialogue to complete, with many fields. I am not going to go into the details of each and every field, however, just take note of key components like:

• Resource Group
  • Note: Meaning you can store all your Alert configurations in a separate Resource Group, and apply RBAC controls to keep people from modifying them.
• Event Category
  • There are currently a total of 6 categories to choose from: Administrator, Security, Service Health, Recommendation, Policy, and Auto Scale. Your selection here will affect the list of Operations that you can choose from.
• Actions
  • You can create an Email Action Group (as I have done here in this screenshot), either for an individual person or even for a group distribution list for a specific team (i.e. IT Security, DBAs, etc.).

NOTE: Per some work done by a fellow MVP (Shamir Charania), the Short Name field has a limitation, requiring a maximum of 12 characters to be used.

Now that I have my Activity Alert configured, let’s test it by shutting down one of my Virtual Machines.

Within a few minutes of triggering the deallocation, I received an email alert as expected.

Note: It’s interesting that the way I configured the Activity Alert (as depicted in the screenshot) actually triggered/fired 3 email alerts. Through investigation, I assume this occurred because my Alert Status was set to “All”, therefore triggering when the deallocation process was initiated, as well as complete (though that does not explain receiving 3 alerts).

Even though I changed the Status setting to “Succeeded” (so as to Alert only when a Virtual Machine was actually deallocated vs. just attempting to do so but failing), I still received 3 email alerts.

However, through additional testing, by setting the Status to “Started” I only received a single email alert. So perhaps this is a bug or a misconfiguration on my part. But keep it in mind (and test it) before you implement it in Production, especially with any integration into your ITSM ticketing solution.

Update: Through yet more testing, it appears that setting the Status to “Succeeded” works as expected, only firing a single email alert. My issue must have been a delay between updating the Alert Rule and triggering the Virtual Machine shutdown/deallocation.

I also discovered why, when setting the Status to “All”, I received 3 email Alerts This is because if you watch the process as it occurs in the Azure Activity Logs, I believe the deallocation process against a Virtual Machine goes from “Started” to “Accepted” to “Succeeded”. As in, the deallocation process is first “Started” (aka triggered through whichever method), and then the underlying Azure Resource Provider (in this case Microsoft Compute) has “Accepted” the process/commend, finally ending in “Succeeded” (assuming the process completed correctly).

Metric Alert

Now let’s briefly look at the Metic Alerts we can configure.

Back under Azure Monitor – Alerts, we have the header as mentioned before. But again, the Add Metric Alert button is disabled/greyed out. This is because Metric Alerts are only applicable to specific resource types (i.e. Virtual Machines).

So in the filter, if we select one of our Azure Virtual Machines, the button will become active, and we can configure a Metric Alert. You can see in the screenshot how it gives you a preview of the metric you are alerting against, to gain an idea if it would trigger, etc.

I personally feel the graph could be enhanced further by including mouse-over details (similar to what we have in other metric reporting tools like OMS).

I also feel this specific configuration approach could be enhanced by allowing us to configure a metric alert rule for a group of computers (i.e. either via Resource Group, Tags, etc.), instead of having to configure it one-by-one. I guess that’s where automation comes into play, eh!

What I also really appreciate is that at the bottom of the Metric Rule blade, we also have the option to take action against the Alert, including running an Automation Runbook, or Logic App.

Also of note, if you choose “Run a Runbook from this Alert”, you can, of course, select your own Azure Automation account, and subsequent Runbook that you would like to initiate. But, there is also a “Standard” source option, in which you can use a Runbook that Microsoft has already created and made available! Nice.

And again, when it fires, we receive the email Alert as expected.

Near Real-Time Metrics Alert

Now let’s look at the Near Real-Time (NRT) Metric Alerts.

The configuration options are similar to the standard Metrics Alerts, but with a few differences.

You will notice that we can now have multiple metrics combined to have more granularity over the Alert. For example, if you want to alert when there is a CPU spike, but also only if there’s a spike in Disk Read bytes at the same time, you can do that.

Currently, this is limited to a maximum of 2 metrics, but hopefully, it will b expanded in the future, allowing even greater flexibility.

Also notice, different from the standard Metrics alert rule, that we can control the evaluation frequency, along with the severity assigned to this rule.

And once again, we receive the email Alert as expected.

Conclusion

In conclusion, Azure Alerts tap into a lot of monitoring areas and has a lot of functionality to fine-tune exactly what you want to be alerted on. As more and more Azure services integrate into Azure Monitor, Azure Alerts will come into play heavily.

But, it’s strongly recommended to test your alerts first to ensure they trigger properly, etc. before integration with ITSM systems, pager systems, etc.

The next tool in our series will be the Azure Diagnostics.