Use Azure Automation for OMS Log Search using REST API


 

OMS is the new player in management space and its gaining momentum every day. In this blog post I want to demonstrate the capabilities around Log Search API using  Azure Automation and how it can help us in daily challenges.

 

Scenario : While managing our datacenters we face many challenges. When we face  a high utilization on a particular server or  slowness  in an application , we need to log into the server and collects/analyze many information to find the root cause.  Operations Manager is already tracking these and reporting  to us but without further  analysis  operator does not have any insight what's is causing the behavior on the target system. In our scenario we will use Azure Automation to  do the investigation using log search API  and send us the results.

 

image

 

General Requirements

First of all you will need an OMS subscription ( you can create one with free tier) and an Azure Automation account  . Free tiers will work perfect for testing. 

Runbook Requirements

 

Authentication

There are different ways to authenticate with Azure Resource Manager (ARM) . In this sample  I’ll be mainly using the ARM REST Api for  almost everything  so I will authenticate using  a Service Principal.  You can also  use native PowerShell cdmlets   to perform the same  . For details you can  check Tiander’s blog;

https://blogs.technet.microsoft.com/privatecloud/2016/04/05/using-the-oms-search-api-with-native-powershell-cmdlets/

To create a  Service principal  for unattended authentication to ARM ;

Go to  Active Directory tab  in Azure Portal

image 

Select Applications

image

Add a new application suing the following option ;

 image

Select Web Application  and give the application an URL.  this URL will be used only for identifying the application so any url will work. For this demo I will use http://myomsclient

image

image

image

On Active Directory Applications page  select  the application you created in last step.

Copy ClientID – This is the username  for our application

Generate a  new key  and select validity 1 or 2 years , this is the password for the service principal.

image

Copy both information as we will be  storing this as variables in Azure automation.

 

Role Assignment in Azure Resource Manager

After service principal is created  it needs to be given permissions to read the information about the  resources  in Azure Resource Manager.  To give Read right on all resources in your subscription  run the following command  in Azure Powershell.

login-AzureRMAccount

New-AzureRmRoleAssignment -RoleDefinitionName Reader –ServicePrincipalName <yourclientid>

To give permission to only OMS Workspace  ;

New-AzureRmRoleAssignment -RoleDefinitionName Reader -ServicePrincipalName <yourclientid>  -ResourceGroupName oi-default-east-us

oi-default-east-us  is the default name for OMS accounts created in East US. You can  run

Get-AzureRmResourceGroup to get the resource group name for your OMS resource group.

image

P.S. Replace ServicePrincipalID and SubscriptionID with your own values

 

Collecting Data

Now we  can authenticate to Azure Resource Manager and start querying OMS Log Search API.

image

For unattended authentication you will need to provide the service principal ID and access key as well as you AD tenant . Easiest way to provide this to the runbook is using Runbook assets . Runbook requires the following assets;

 

$SPAppID = Get-AutomationVariable -Name 'SPCertAppID'    # Variable Asset   - ClientID of the AD Application
$subscriptionid= Get-AutomationVariable -Name 'MsIntSubsID' # Variable Asset   -  Azure subscriptionID
$CLIENTKEY=Get-AutomationVariable -Name 'ArmclientKey'  # Variable Asset   -  Key to authenticate with the AS application
$resgrp=Get-AutomationVariable -Name 'OIEUSRG'     #Variable Asset   - OMS resource group   oi-default-east-us  in my case
$oiwssclab=Get-AutomationVariable -Name 'OIWS_SCLAB'     #Variable Asset   - OMS workspace Name
$hotmailcreds=Get-AutomationPSCredential -Name "myhotmail" # Credential Asset  : my Hotmail  credential
$mailto=Get-AutomationVariable -Name "Notificationrecp" #Variable Asset   -  Receipent email address
$tenant = Get-AutomationVariable -Name "ADTenant" #Variable Asset   - My Ad tenant name
$ARMUrl = "https://management.azure.com"
$ARMGraphUrl = "https://graph.windows.net"
$apiver=Get-AutomationVariable -Name 'ArmApiVersion'   # Variable asset    . I’m using 2015-03-20    as API Version

Complete runbook can be downloaded from ;

 

Performing Queries

Runbook utilizes search-oi function to perform the OMS search.  To make your own search  first define the query

$query="Type=SecurityEvent AccountType=user (EventID=4624)   AND TimeGenerated>NOW-8HOUR (Computer=""$server"") LogonTypeName=""2 - Interactive"" | top 5|select Account, Process,timegenerated"

and supply it as a parameter to search-oi function

     $interactivelogons=search-oi -query $query

Query result :

image

By using these technique you can modify/extend the searches performed in this runbook.

 

Email Findings

Runbook  will get all returned search queries and construct a html email body  .  We will send this mail to the recipient  using the Hotmail credentials . Here after using the code Hotmail will mpst probably will block it as suspicious activity. You need to go the  settings and allow this activity. If you use other email provider you might need to  enable some security so Azure Automation can authenticate and send email on your behalf.

 

After collecting the data I’ll send  the findings using the following code

$SMTPServer = "smtp.live.com"
$SMTPPort = "25"
$Username = "yourliveid@hotmail.com"
$Password = ""

$to = "<yourrecipientaddress>"

$subject = "OMS Search Demo"

$message = New-Object System.Net.Mail.MailMessage
$message.subject = $subject
$message.body = $body
$message.to.add($to)

$message.from = $username

$message.IsBodyHTML=$true

$message.AlternateViews
$smtp = New-Object System.Net.Mail.SmtpClient($SMTPServer, $SMTPPort);
$smtp.EnableSSL = $true
$smtp.Credentials = $hotmailcreds

$smtp.send($message)
write-output  "Mail Sent"

Webhook

Since we will be  triggering this runbook  from OMS/OpsMgr    easiest way to implement it  is to use Webhooks.

Webhooks are URLs which we can utilize to  trigger the runbook. All security is based on  the URL so when you create the webhook keep the URL safe.

Navigate to your automation Account / runbooks select  the runbook you have created and then select Webhooks

image

Create a new webhook

image

Give webhook a name,  enable it and set an end date. Here you need to copy  and save the URL  , after saving the webhook you wont be able to fetch it again!!!

From webhook run settings select Run on Azure  and save .

Now you will be able to trigger this runbook by Http POST.

 

Operations Manager Notification Channel

Now we will utilize this webhook  to sstart log search automation by using the following script

Param ($server,$alert,$data )

$uri = “https://s1events.azure-automation.net/webhooks?token=replacewithyourownWebhookURI
$headers = @{"From"=Scom@contoso.local;"Date"="$((Get-date).ToString())"}

$vms  = @(
            @{ Server="$server";Alertname="$alert";Alertdata="$data";}
           
        )
$body = ConvertTo-Json -InputObject $vms

$response = Invoke-RestMethod -Method Post -Uri $uri -Headers $headers -Body $body

 

IN Operations Manager  you can trigger this  webhook using the  powershell script above in a command notification channel ;

 

image

"G:\MPauthoring\OMSNotification.ps1" -server '$Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$' -alert  '$Data[Default='Not Present']/Context/DataItem/AlertName$' -data '$Data[Default='Not Present']/Context/DataItem/AlertDescription$'

So all you need is to create a notification subscription  using this new channel  for the selected alerts and  Operations Manager will trigger  the runbook  Please keep in mind,  for  this scenario to work your Management server needs to be able to  access the webhook url.

 

Result

 

Here is the  email  generated after the automated search. I have queried almost all  the data I have for that specific server in this demo. In real life you can decide which details to fetch from OMS based on your own needs.

image

image

image

 

 

Happy automating and utilizing OMS !

Skip to main content