Using the HTTP OMS Data Collector API for real-world scenario’s–Part 2


In the first part of this blog post series I started with a PowerShell function to retrieve stock information and converting this to the correct format so the Log Analytics HTTP Data Collector API is able process the record with stock information.

For more information about how you would post JSON data to the Log Analytics repository from any client able to call the REST API I refer to the Azure documentation here: https://azure.microsoft.com/en-us/documentation/articles/log-analytics-data-collector-api/

On that page you can find a PowerShell example which we are going to use as basis for our Azure Automation Runbook which will do the following:

  • Retrieve stock information
  • Format retrieved stock data so the Log Analytics HTTP Data Collector API is able process the record
  • Create PowerShell Functions to create a authorization signature and finally send the data to OMS.

StockPriceOMS Runbook:

If you are new to creating Azure Automation Runbooks I would recommend that you have a look at some my earlier blog post about Azure Automation and Runbooks. Another great start is the Azure Automation overview from Matt Goedtel .

I created the following PowerShell script which we are going to use for the Azure Automation Runbook.

[sourcecode language='bash'  padlinenumbers='true']
#requires -version 3.0

# ---------------------------------------------------
# Script: C:\Users\Stefan\Documents\GitHub\AzureAutomationDemo\StockPriceOMS.ps1
# Tags: OMS, PowerShell, Azure, Runbook, Automation
# Version: 0.1
# Author: Stefan Stranger
# Date: 09/05/2016 17:07:24
# Description: Demo Runbook which retrieve stock information and uses the Log Analytics HTTP Data Collector API
#              to send this data to OMS. 
# Comments:
# Changes:  
# Disclaimer: 
# This example is provided “AS IS” with no warranty expressed or implied. Run at your own risk. 
# **Always test in your lab first**  Do this at your own risk!! 
# The author will not be held responsible for any damage you incur when making these changes!
# ---------------------------------------------------

#region variables
# Replace with your Workspace ID
$CustomerID = Get-AutomationVariable -Name "CustomerID"
# Replace with your Primary Key
$SharedKey = Get-AutomationVariable -Name "SharedKey"
#Specify the name of the record type that we'll be creating.
$LogType = Get-AutomationVariable -Name "Logtype"
#Specify a time in the format YYYY-MM-DDThh:mm:ssZ to specify a created time for the records.
$TimeStampField = Get-AutomationVariable -Name "Logtype"
#Specify the name of the Stock we want to retrieve.
$StockName = Get-AutomationVariable -Name "StockName"
#Create Array
$StockName = $StockName.Split(",")
#endregion

#region functions
# Function to retrieve Stock value information
function Get-StockPrice
{
  [CmdletBinding()]
  Param
  (
    # Stockname(s)
    [Parameter(Mandatory = $true,
        ValueFromPipelineByPropertyName = $true,
    Position = 0)]
    $StockName
  )

  Process
  {
    $HttpRequestUrl = 'http://finance.google.com/finance/info?client=ig&q={0}'-f $StockName

    $stockvalue = Invoke-WebRequest -Uri $HttpRequestUrl -UseBasicParsing

    #Fixing output. ONLY works with one STOCK PRICE!
    $stockprice = ((($stockvalue.content).Replace('//','')).Replace('[','')).Replace(']','') | ConvertFrom-Json

    #Convert to correct types

    $stockprice.id = [int]$stockprice.id
    $stockprice.l = [double]$stockprice.l
    $stockprice.l_fix = [double]$stockprice.l_fix
    $stockprice.l_cur = [double]$stockprice.l_cur
    $stockprice.c = [double]$stockprice.c
    $stockprice.c_fix = [double]$stockprice.c_fix
    $stockprice.cp = [double]$stockprice.cp
    $stockprice.cp_fix = [double]$stockprice.cp_fix
    $stockprice.pcls_fix = [double]$stockprice.pclx_fix

    ConvertTo-Json -InputObject @($stockprice)

  }
}

# Function to create the authorization signature.
Function New-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
  $xHeaders = 'x-ms-date:' + $date
  $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource

  $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
  $keyBytes = [Convert]::FromBase64String($sharedKey)

  $sha256 = New-Object -TypeName System.Security.Cryptography.HMACSHA256
  $sha256.Key = $keyBytes
  $calculatedHash = $sha256.ComputeHash($bytesToHash)
  $encodedHash = [Convert]::ToBase64String($calculatedHash)
  $authorization = 'SharedKey {0}:{1}' -f $customerId, $encodedHash
  return $authorization
}

# Function to create and post the request
Function Send-OMSData($customerId, $sharedKey, $body, $logType) 
{
  $method = 'POST'
  $contentType = 'application/json'
  $resource = '/api/logs'
  $rfc1123date = [DateTime]::UtcNow.ToString('r')
  $contentLength = $body.Length
  $signature = New-Signature `
  -customerId $customerId `
  -sharedKey $sharedKey `
  -date $rfc1123date `
  -contentLength $contentLength `
  -fileName $fileName `
  -method $method `
  -contentType $contentType `
  -resource $resource
  $uri = 'https://' + $customerId + '.ods.opinsights.azure.com' + $resource + '?api-version=2016-04-01'

  $headers = @{
    'Authorization'      = $signature
    'Log-Type'           = $logType
    'x-ms-date'          = $rfc1123date
    'time-generated-field' = $TimeStampField
  }

  $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
  return $response.StatusCode
} 
#endregion

# Main
# Submit the data to the API endpoint
# Iterate through the configured StockNames.
Foreach($Stock in $StockName) {
    $json = Get-StockPrice -StockName $Stock
    Write-Output $json
    Send-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $logType 
}
[/sourcecode]

Explanation of Runbook PowerShell script:

This PowerShell Script Runbook has the following sections:

  • Azure Automation Assets
  • Three Functions
    • Get-StockPrice
    • New-Signature
    • Send-OMSData

If you look at the Functions you see they have some parameters that need to be configured using Azure Automation Assets. The following Azure Automation Assets need to be configured:

  • CustomerID (OMS Workspace ID, see picture below)
  • SharedKey (Primary Key, see picture below)
  • LogType (this is name of the Type of data entries you want to use for the Stock data entries. This could be anything, like “StockPrice”. Keep in mind OMS will ad “_CL” to your LogType Name)
  • TimeStampField (Specify a time in the format YYYY-MM-DDThh:mm:ssZ to specify a created time for the records.) See here for more info). Not configured in the example.
  • StockName (this is string with one of more stock names separated by a comma. Example: “MSFT,GOOGL” or “MSFT”)

You can find the CustomerID aand SharedKey information within the OMS Portal under Overview –> Settings –> Connected Sources

image

 

Configure Azure Automation Assets in the Azure Portal

Go to Azure Portal, select your Automation Account and select Assets.

image

Create the following Variables Assets with your own values. All the Variable Assets are of the type String but make sure you select Encrypted for the CustomerID and SharedKey Variable Assets because this is confident information you would keep kind of secured.

image

 

Create StockPrice Runbook

Go to the Azure Portal, select your Automation Account and select Runbooks.

image

Click on Add Runbook –> Create a new runbook.

image

Enter a Name and select Runbook type PowerShell and click on Create.

Copy and paste PowerShell script from above to editor in the Azure Portal and click on Save.

image

Before publishing the Runbook we are first going to test the Runbook. Select the Test Pane button and click on Start.

image

If the test is successful you should see something similar to below example.

 image

We can now publish the Runbook. We need to Publish the Runbook if we want to add a Schedule to have the Runbook running at a configured interval, like every hour.

image

Publish the Runbook and let’s run the Runbook manually  a couple of times to get some more data collected and send to OMS.

Check StockPrice data in OMS

Open your OMS workspace

image

Select OMS Portal to open the portal.

image

Go to the Log Search Solution to search for the StockPrice_CL Type entries.

image

Depending on the name you configured earlier for the LogType in the Assets you can search for StockPrice_CL. (OMS adds  _CL to the name of your configure LogType variable)

image

For more info on the field name of the StockPrice data being send to OMS please check the documentation.

The final step in this part of the blog post series is configuring a Schedule for the StockPrice Runbook.

Runbook Schedule

Now we have the Runbook tested and published we want to run this runbook regarly, say every hour. For this to happen we need to add a Schedule to the StockPriceOMS Runbook in the Azure Automation Account Portal.

Go to Automation Accounts -> AutomationAccount -> Runbooks -> Runbook –> Settings and click on Schedules.

 

image

 

Select Add Schedule and create a new Schedule.

image

image

Enter the values to have the Schedule run every hour.

image

If you have configured the Schedule correctly the StockPrice Runbook will run every hours and StockPrice data will be send to OMS.

Watch this blog for the next blog post in this series of blog posts. In the next blog post we are going to create an OMS alert to get notified when a certain threshold is reached.


Comments (2)

  1. Josh Burkard says:

    when i run this runbook, the Runbook Output will be like above. But on OMS i will see only two fields: TimeGenerated and SourceSystem. In Field SourcSystem thee is the Value “RestAPI”

    1. Scott says:

      Did you manage to find the problem to this? I’m having the same issue with another script and can’t figure out why it’s not showing me all of the fields

Skip to main content