Alerts from Azure Automation

in a previous post, i showed how to set up statistics maintenance for your azure databases using azure automation . however, what i didn’t show was how to generate an alert when things go south. let’s do that now.

an error needing an alert

first, i need to generate an error. i’m going to modify the code just slightly from the previous example so that it will fail:

$cmd=new-object system.data.sqlclient.sqlcommand("update statistics dbo.tablenotindb", $conn)

if i modify my runbook with the code above and then run it, i will get an error:

this is from the test pane. i strongly recommend you use tests on your powershell scripts when writing your own automation. it’ll save you a lot of pain trying to troubleshoot things later.

alert on error

believe it or not, there’s not an immediately obvious “oh, you had an error in your automation script, here’s how you alert someone” setting in the azure portal. now, you could simply put error handling in your powershell script. in fact, it’s probably not at all a bad idea to do that as well. however, what you would not get setting things up that way is a mechanism for managing the alerts, history, additional possible responses (like firing off another runbook, although there is a way to do that from the powershell, too). instead, what i want is a way to manage alerts through the azure fabric.

if you do a search, there is an azure alert service. however, it didn’t seem to be really what i was looking for. furthermore, i found it extremely difficult (ok, i couldn’t make it work) to connect the alerts directly to the jobs related to my runbooks. instead, after quite a bit of research, what i found is a combination of azure log analytics with the operations management suite (oms) will do exactly what i’m looking for.

on a personal note, i kind of can’t believe i’m working with the operations manager again. i was writing articles about that at the very start of my time as an mvp 10 years ago. wild how things come around again.

getting started with alerts

first things first, you have to already have set up your azure automation to put this all together. then, you need to create a log analytics workspace. when you have those done, and, assuming you’ve got your powershell all up to date (that was the single longest part of making all this work), you can follow the directions in this nice microsoft blog post to connect them together.

caution: there are a bunch of posts out there purporting to show you how to connect these together using powershell, but they’re mostly over a year old, see my blog post from yesterday .

the key command is set-azurermdiagnosticsetting. pass in the correct values from your automation and log analytics and the rest of this more or less falls into place.

at this point, i stepped out of the code and went over to operations manager. you should be automating everything you do, and following along with the microsoft blog post to set up and manage your system using powershell is the correct approach. however, let’s look at pretty pictures:

that’s the default page for the oms.

to set up an alert is pretty simple. first, let’s do a search of the logs to see if we successfully have everything connected. for my purposes, i have my automation script (which is failing) running a few times an hour so that i don’t have to wait to see results. the basic log search looks like this:

you can type in a * wild card search or just click on the link that will show all data collected (only do that when you’re getting started, not when you have lots of systems under management). scrolling through the results, i can find where my automation job failed and generated an error:

with this, i can create a new filter, one based on the data i have. instead of *, i’ll use this:

runbookname_s=updatestatistics streamtype_s=error

it’s almost like a t-sql query except no requirements for and or quotation marks for the string values. that will filter the results. all i have to do now is click the alert button you’ll see on the screen and i’m setting up my alert:

the rest is pretty self-explanatory (click on the image to make it larger & more visible). go through and fill out the fields and save this alert. you’ll be getting emails in no time.

conclusion

the beauty of azure, and the pain of azure, is that there is usually more than one way to get something done. my intention with this approach was to stick to services rather than resort to code-based solutions. yes, code to manage those solutions, but not simply code to run things. this way, you get all the features that the service offers, not simply what you’ve coded.