Automation–Windows Azure Pack & Service Management Automation–Automated Tenant Deprovisioning

Hello Readers!

Today’s topic is one that has been coming up more and more as the usage of Windows Azure Pack (WAP) increases.


Problem Statement / Question / Answer

Problem “When I delete the Subscription for a WAP User (or the actual User), their Subscription Resources remain in VMM!”

Question “How do I automatically remove Tenant Subscription Resources after the Subscription (or User and Subscription) has been removed from WAP?”

Answer Service Management Automation (SMA) Runbooks constructed specifically to remove Subscription Resources, tied to the WAP/SPF Subscription.Delete Action Event.


Background

At the beginning of this year, I published a series of posts around Automated Tenant Provisioning. At that time, it was far more important to get something published that had to do with the creation of resources within WAP, rather than the other side of the coin: removal of those resources within WAP. And while I always had scripts that would remove a given set of resources (who wants to manually do something that can be scripted!?), I never made them “official” – even in my Dev/Test/Prod environments.

Recently, while working on a project where literally hundreds of subscriptions were being created with then intention of immediate removal, I decided to turn my “Deprovisioning Scripts” into actual “Tenant Deprovisioning Runbooks”. I even expanded on their original functionality, linking them to action events within WAP/SPF, and adding even more Deprovision functionality (covering most all of my daily Deprovisioning scenarios).


Scope

So, this blog post is how I went about answering the above question – within a certain scope, of course. There are many ways to accomplish what I am about to provide examples for – so, please leverage this content accordingly.

Here is the scope of this post:

  • Removal of Tenant Subscription Resources based on Subscription (or User + Subscription) removal
  • Removal is based on Subscription ID (it is the parameter passed into the Process Runbook)
  • Removal of the following Subscription Resources:
    • VMM Cloud Services (Which for my environment, based on my previous examples, contain all deployed VMM Cloud Resources, or VMRoles)
    • VMM Standalone VMs (VM Template deployed Virtual Machines)
    • VMM VM Networks (Including Removal of all network specific dependencies (e.g. Gateways, Subnets, etc.)
    • VMM User Role (While this usually gets removed when a Subscription is deleted, I have it as part of my process to ensure its removal)
    • WAP Subscription (Again, while this is usually removed as part of the Subscription deletion in the portal, I have seen instances where a second call to this via PowerShell with -Force is required)
  • Removal of the WAP User (A bonus example Runbook I have used to initiate the entire process during load/unload testing – this can be called within a foreach to perform automated Deprovisioning at a large scale)
  • Removal of the VMM Cloud Resources (Another bonus example Runbook I used before I discovered the over-arching Cloud Service Removal – this may or may not be useful)
  • Subscription-Delete-Dispatcher (The SPF tagged Runbook that gets invoked whenever the WAP/SPF Subscription.Delete Action Event fires
  • Example Set/Create Script to ensure the required SMA Assets (Variables and Credentials) can easily be created

Prerequisites

This set of examples is not without its prerequisites. The following is a list of setup/configuration needed to ensure these examples work in your environment:

These may seem like steep prerequisites, but really, they are the same as the Tenant Provisioning Scenario, of which you likely already have working (or at least something similar), otherwise you wouldn’t be looking for this set of examples.


Process

The Deprovisioning process is simple compared to the Provisioning process. In these examples I go directly against VMM (and WAP where appropriate) , using the well known Cmdlets.

The following is the high-level process:

image


Runbooks

Now for what you all have been waiting for (or what you skipped the first 4 sections for) … ;)

This post contains the following Runbooks:

image

Also listed here as text with their nesting associations:

  • Subscription-Delete-Dispatcher 
         --> Get-WAPSubscription
         --> Remove-SubscriptionResources
              --> Remove-VMMCloudService
              --> Remove-VMMStandaloneVM
              --> Remove-VMMVMNetwork
              --> Remove-VMMUserRole
              --> Remove-WAPSubscription
  • Bonus! Runbooks
    - Remove-VMMCloudResource
    - Remove-WAPUser

Note These Runbooks EXAMPLES are by no means perfect. But they are what I use every day in hundreds of various Deprovisioning jobs.

Disclaimer As the name indicates, these example scripts are meant to Deprovision resources from an environment. They are DESTRUCTIVE. Please ensure you either these as guidance, or have modified them in such a way that works best for your environment. Remember, these are here to provide an automated example to save you clicks. You do not have to use them. Also, they could stand some more Error Handling, I get that. I wanted to get them out as soon as possible, so that you could consume, digest, and refactor as you see fit.


Core Example Runbook Scripts

And now for the actual scripts. I will be displaying them in the same order as listed immediately above.

Subscription-Delete-Dispatcher

001002003004005006007008009010011012013014015016017018019020021022023024025026027028029030 workflow Subscription-Delete-Dispatcher{ param ( [object]$resourceObject ) #Get WAP Info and Token $WAPServer = Get-AutomationVariable -Name 'WAP Server' $ClientRealm = Get-AutomationVariable -Name 'Default Client Realm' $WAPCreds = Get-AutomationPSCredential -Name 'WAP Credentials' $Token = Get-Token -WAPServer $WAPServer -clientRealm $ClientRealm -Creds $WAPCreds #Get VMM Info $VmmServerName = Get-AutomationVariable -Name 'VMM Server' $VMMCreds = Get-AutomationPSCredential -Name 'VMM Credentials' #Get Subscription $Subscription = Get-WAPSubscription -SubscriptionID $resourceObject -WAPCreds $WAPCreds ` -WAPServer $WAPServer -Token $Token if (!$Subscription.Error -and $Subscription.SubscriptionObj.State -eq "DeletePending") { Write-Output "Removing all Resources associated with SubscriptionID $resourceObject" Remove-SubscriptionResources -SubscriptionID $resourceObject ` -VMMCreds $VMMCreds -VMMServerName $VmmServerName ` -WAPServer $WAPServer -WAPCreds $WAPCreds -Token $Token } else { Write-Output "$resourceObject No Longer Exists" }}

Notes

  1. This Runbook example contains a few SMA Assets (Variables and Credentials). The requirements for these can be found in the next section titled, “Runbook Variable and Credential Asset Requirements”
  2. This Runbook example references a Sub-Runbook named “Get-Token” – it is not part of these examples, but has been documented/described previously (see this post , search for “Get and Usage of the WAP MgmtSvcToken”)
  3. Depending on the authentication method in your environment, you may need to get an ADFS Token – again, it is not part of these examples, but has also been documented/described previously (see this post , search for “Requesting a token with ADFS”)
  4. Because the Delete.Subscription Action Event initiates every time a Subscription is marked for delete (even if it is done multiple times, and even if it is done while in “Deletion in Progress” status), it was important to add the Get-WAPSubscription and include logic which could determine whether or not the Subscription still existed or not
  5. Because Removal of the Subscription happens as part of the process (inside the Remove-SubscriptionResources Runbook), the Subscription-Delete-Dispatcher will initiate twice for every Subscription deleted (this is an extra measure that I took due to some issues with deletion from time to time).

Get-WAPSubscription

001002003004005006007008009010011012013014015016017018019020021022023024025026027028029030031032033034035036037038039040041 workflow Get-WAPSubscription{ param ( [string]$SubscriptionID, [string]$WAPServer, [PSCredential]$WAPCreds, [string]$Token ) #Get WAP Connection Info $AdminUri = "https://{0}:30004" -f $WAPServer #Connect to WAP and Get Subscription Info InlineScript { try { $ErrorActionPreference = "Stop" $WAPSubscription = Get-MgmtSvcSubscription -AdminUri $Using:AdminUri ` -SubscriptionId $Using:SubscriptionID ` -Token $Using:Token } catch { $errorMessage = $Error[0].Exception } finally { $ErrorActionPreference = "Continue" } $SubStatusMsg = [PSCustomObject]@{ SubscriptionID = $using:SubscriptionID SubscriptionObj = $WAPSubscription Error = $errorMessage } Write-Output $SubStatusMsg } -PSComputerName $WAPServer -PSCredential $WAPCreds}

Note This Runbook contains more error handling than I normally provide in my examples. Based on how the WAP Cmdlets handle “Gets” when something doesn’t exist, this was my best option. Remember, this is just an example. There are other ways to handle this.


Remove-SubscriptionResources

001002003004005006007008009010011012013014015016017018019020021022023024025026027 workflow Remove-SubscriptionResources{ param ( [object]$SubscriptionID, [string]$VmmServerName, [PSCredential]$VMMCreds, [string]$WAPServer, [PSCredential]$WAPCreds, [string]$Token ) Remove-VMMCloudService -SubscriptionID $SubscriptionID -VMMCreds $VMMCreds ` -VMMServerName $VmmServerName Remove-VMMStandaloneVM -SubscriptionID $SubscriptionID -VMMCreds $VMMCreds ` -VMMServerName $VmmServerName Remove-VMMVMNetwork -SubscriptionID $SubscriptionID -VMMCreds $VMMCreds ` -VMMServerName $VmmServerName Remove-VMMUserRole -SubscriptionID $SubscriptionID -VMMCreds $VMMCreds ` -VMMServerName $VmmServerName Remove-WAPSubscription -SubscriptionID $SubscriptionID -WAPCreds $WAPCreds ` -WAPServer $WAPServer -Token $Token}

Note This is a simple Process Runbook. It is meant to hold the logic (ordering mostly) and calls out to the various Sub-Runbooks. As you can see it takes in and leverages both WAP and VMM information.


Remove-VMMCloudService

001002003004005006007008009010011012013014015016017018019020021022023024025 workflow Remove-VMMCloudService{ param ( [string]$SubscriptionID, [string]$VmmServerName, [PSCredential]$VMMCreds ) InlineScript { Get-SCVMMServer -ComputerName $Using:VmmServerName | Out-Null $CloudServices = Get-CloudService | ? {$_.UserRoleID -eq $Using:SubscriptionID} if ($CloudServices) { foreach ($CloudService in $CloudServices) { $CloudServiceRemove = Remove-CloudService -CloudService $CloudService Write-Verbose $CloudServiceRemove $CSRemoveMsg = "Removing {0} {1} for {2}" -f $CloudServiceRemove.ObjectType,$CloudServiceRemove.Name,$CloudServiceRemove.UserRole Write-Output $CSRemoveMsg } } else { Write-Output "No Cloud Services exist for Subscription: $Using:SubscriptionID" } } -PSComputerName $VmmServerName -PSCredential $VMMCreds}

Note If you followed my guidance for Automated Tenant Provisioning, you likely have all your VMRoles (Cloud Resources) leveraging the same Cloud Service (naming convention: CloudService-4-<SubscriptionGUID>). If this is the case, you only need to remove the one Cloud Service containing all your VMRoles. If not, it will still get and remove all Cloud Services associated Cloud Resources.


Remove-VMMStandaloneVM

001002003004005006007008009010011012013014015016017018019020021022023024025026027028029030031 workflow Remove-VMMStandaloneVM{ param ( [string]$SubscriptionID, [string]$VmmServerName, [PSCredential]$VMMCreds ) InlineScript { Get-SCVMMServer -ComputerName $Using:VmmServerName | Out-Null $StandaloneVMs = Get-SCVirtualMachine | ? {$_.UserRoleID -eq $Using:SubscriptionID -and $_.CloudVmRoleName.Length -eq 0} if ($StandaloneVMs) { foreach ($StandaloneVM in $StandaloneVMs) { $StandaloneVMStop = Stop-SCVirtualMachine -VM $StandaloneVM -Force Write-Verbose $StandaloneVMStop $SAVMStopMsg = "Stopping {0} {1} for {2}" -f $StandaloneVMStop.ObjectType,$StandaloneVMStop.Name,$StandaloneVMStop.UserRole Write-Output $SAVMStopMsg $StandaloneVMRemove = Remove-SCVirtualMachine -VM $StandaloneVM Write-Verbose $StandaloneVMRemove $SAVMRemoveMsg = "{0} {1} {2} for {3}" -f $StandaloneVMRemove.VirtualMachineState,$StandaloneVMRemove.ObjectType,$StandaloneVMRemove.Name,$StandaloneVMRemove.UserRole Write-Output $SAVMRemoveMsg } } } -PSComputerName $VmmServerName -PSCredential $VMMCreds}

Note This example includes both the Stop and Remove commands required to delete a Standalone VM from VMM. I am also identifying this VM object as non-VMRole by checking that the CloudVmRoleName.Length -eq 0. There are other ways to do this, this is the way that worked for me.


Remove-VMMVMNetwork

001002003004005006007008009010011012013014015016017018019020021022023024025026027028029030031032033034035036037038039040041042043044045 workflow Remove-VMMVMNetwork{ param ( [string]$SubscriptionID, [string]$VmmServerName, [PSCredential]$VMMCreds ) InlineScript { Get-SCVMMServer -ComputerName $Using:VmmServerName | Out-Null $VMNetworks = Get-SCVMNetwork | where {$_.UserRoleID -eq $Using:SubscriptionID} if ($VMNetworks) { foreach ($VMNetwork in $VMNetworks) { $VMNGW = Get-SCVMNetworkGateway -VMNetwork $VMNetwork | Remove-SCVMNetworkGateway Write-Verbose $VMNGW $VMNGWRemoveMsg = "Removing {0} {1} for SubscriptionID: {2}" -f $VMNGW.ObjectType,$VMNGW.Name,$Using:SubscriptionID Write-Output $VMNGWRemoveMsg $VMSubnet = Get-SCVMSubnet -VMNetwork $VMNetwork $VMNIPPool = Get-SCStaticIPAddressPool -VMSubnet $VMSubnet | Remove-SCStaticIPAddressPool Write-Verbose $VMNIPPool $VMNIPPRemoveMsg = "Removing {0} {1} for SubscriptionID: {2}" -f $VMNIPPool.ObjectType,$VMNIPPool.Name,$Using:SubscriptionID Write-Output $VMNIPPRemoveMsg $VMNSubnet = Remove-SCVMSubnet $VMSubnet Write-Verbose $VMNSubnet $VMNSRemoveMsg = "Removing {0} {1} for SubscriptionID: {2}" -f $VMNSubnet.ObjectType,$VMNSubnet.Name,$Using:SubscriptionID Write-Output $VMNSRemoveMsg $VMN = Remove-SCVMNetwork $VMNetwork Write-Verbose $VMN $VMNRemoveMsg = "Removing {0} {1} for SubscriptionID: {2}" -f $VMN.ObjectType,$VMN.Name,$Using:SubscriptionID Write-Output $VMNRemoveMsg } } } -PSComputerName $VmmServerName -PSCredential $VMMCreds}

Note This example is the most involved of all the Deprovisioning Runbooks. It contains the most moving parts, and is the most “delicate”. You likely know what it takes to create a network with all the various options, so tearing that network back down has to happen in a particular order. This example will remove the following, in the following order: Gateway, Static IP Address Pool, Subnet, and finally the Network. Your environment will vary, so be sure to modify to fit.


Remove-VMMUserRole

001002003004005006007008009010011012013014015016017018019020021022 workflow Remove-VMMUserRole{ param ( [string]$SubscriptionID, [string]$VmmServerName, [PSCredential]$VMMCreds ) InlineScript { Get-SCVMMServer -ComputerName $Using:VmmServerName | Out-Null $VMMUserRole = Get-SCUserRole -ID $Using:SubscriptionID if ($VMMUserRole) { $VMMUserRoleRemove = Remove-SCUserRole -UserRole $VMMUserRole Write-Verbose $VMMUserRoleRemove $URRemoveMsg = "Removing {0} for {1}" -f $VMMUserRoleRemove.ObjectType,$VMMUserRoleRemove.Name Write-Output $URRemoveMsg } } -PSComputerName $VmmServerName -PSCredential $VMMCreds}

Note This example is me being (likely) overly thorough. I believe I noticed that even though the Subscription Delete in WAP is supposed to Remove the VMM User Role, sometimes it did not. To ensure that it is deleted, I added this Sub-Runbook to the process. It makes sure the VMM User Role does not remain once a WAP Subscription Removal has been initiated.


Remove-WAPSubscription

001002003004005006007008009010011012013014015016017018019020021 workflow Remove-WAPSubscription{ param ( [string]$SubscriptionID, [string]$WAPServer, [PSCredential]$WAPCreds, [string]$Token ) #Get WAP Connection Info $AdminUri = "https://{0}:30004" -f $WAPServer #Connect to WAP and Remove Subscription InlineScript { Remove-MgmtSvcSubscription -AdminUri $Using:AdminUri -SubscriptionId $Using:SubscriptionID ` -Token $Using:Token -Confirm:$false -Force } -PSComputerName $WAPServer -PSCredential $WAPCreds}

Note Again, this is me being (likely) overly thorough. Sometimes, it appears that the Subscription gets “stuck” in “Deletion Pending” (or something similar) – this ensures that it gets removed (the -force works wonders in the WAP Cmdlets).


Bonus! Example Runbook Scripts

Remove-VMMCloudResource

001002003004005006007008009010011012013014015016017018019020021022023024025 workflow Remove-VMMCloudResource{ param ( [string]$SubscriptionID, [string]$VmmServerName, [PSCredential]$VMMCreds ) InlineScript { Get-SCVMMServer -ComputerName $Using:VmmServerName | Out-Null $CloudResources = Get-CloudResource | ? {$_.UserRoleID -eq $Using:SubscriptionID} if ($CloudResources) { foreach ($CloudResource in $CloudResources) { $CloudResourceRemove = Remove-CloudResource -CloudResource $CloudResource Write-Verbose $CloudResourceRemove $CRRemoveMsg = "{0} {1} {2} for {3}" -f $CloudResourceRemove.ProvisioningState,$CloudResourceRemove.ObjectType,$CloudResourceRemove.Name,$CloudResourceRemove.UserRole Write-Output $CRRemoveMsg } } else { Write-Output "No Cloud Resources exist for Subscription: $Using:SubscriptionID" } } -PSComputerName $VmmServerName -PSCredential $VMMCreds}

Note As noted above in the Remove-VMMCloudService example, this script is likely not necessary. It also is nearly identical to the Cloud Service example, with a few keywords swapped. It may be useful for out-of-band Cloud Resource Removals (orphaned resources, etc.).


Remove-WAPUser

001002003004005006007008009010011012013014015016017018019020021022023024025026027028029030 workflow Remove-WAPUser{ param ( [string]$User, [boolean]$DeleteSubscriptions ) #Get Token $WAPServer = Get-AutomationVariable -Name 'WAP Server' $ClientRealm = Get-AutomationVariable -Name 'Default Client Realm' $WAPCreds = Get-AutomationPSCredential -Name 'WAP Credentials' $Token = Get-Token -WAPServer $WAPServer -clientRealm $ClientRealm -Creds $WAPCreds #Get WAP Connection Info $AdminUri = "https://{0}:30004" -f $WAPServer #Connect to WAP and Remove Subscription InlineScript { if ($Using:DeleteSubscriptions -eq $true) { Remove-MgmtSvcUser -AdminUri $Using:AdminUri -Name $Using:User ` -Token $Using:Token -Confirm:$false -DeleteSubscriptions } else { Remove-MgmtSvcUser -AdminUri $Using:AdminUri -Name $Using:User ` -Token $Using:Token -Confirm:$false -Force } } -PSComputerName $WAPServer -PSCredential $WAPCreds}

Note This is meant to be a standalone Runbook that can be executed on its own (thus the specific callout to Get a Token). It can be used to force the deletion of a stubborn WAP User (one that gets stuck in “Deletion in Progress” (or something similar)) or to clear out a User and Subscription. Either way, if this is used, and the Subscription.Delete Action Event is hooked to a Runbook, that Runbook will be fired. If it is hooked to the set above, then all the associated resources will be removed as well.


Assets

As you can see in the above examples, I have leveraged a few Assets (Variables and Credentials).

Runbook Variable and Credential Asset Requirements

The following is a listing of the required Runbook Variable and Credential Assets.

Variable Assets

  • WAP Server - Name of the WAP Server (or at least one node of WAP where the Cmdlets are installed)
  • Default Client Realm - In everything I do for all these examples: https://azureservices/AdminSite
  • VMM Server - Name of the VMM Server (either one of the nodes, or the cluster name)

Credential Assets

  • WAP Credentials - Credentials with permissions to perform the remove actions in WAP
  • VMM Credentials - Credentials with permissions to perform the remove actions in VMM

Note These Variables and Credentials are primarily used to store information about the environment (WAP and VMM) and the ability to access/authenticate to perform the above scripts. The Token is used to authenticate against the SM API; and the WAP/VMM Credentials are used to remotely connect to the WAP and VMM targets to perform the commands within the scripts.


Creating the SMA Assets

One of the best ways that I have found to make the SMA Assets portable for a given set of Runbooks is to create a “Set Script”. This script leverages the very versatile Set-SmaVariable Cmdlet found in the SMA Module. This Cmdlet has the ability to both SET existing and CREATE new Assets.

The following is an example “Set Script” which can be modified and executed to create the Variable and Credential Assets required for these Deprovisioning Runbooks:

001002003004005006007008009010011012013014015 #SET Variable for SMA Web Service Endpoint$smaWSEndpoint = "https://sma-server.fqdn"#SET Variable AssetsSet-SmaVariable -Name "Default Client Realm" -Value "https://azureservices/AdminSite" -WebServiceEndpoint $smaWSEndpointSet-SmaVariable -Name "VMM Server" -Value "vmm-server.fqdn" -WebServiceEndpoint $smaWSEndpointSet-SmaVariable -Name "WAP Server" -Value "wap-server.fqdn" -WebServiceEndpoint $smaWSEndpoint#SET Credential Assets (by prompt)Set-SmaCredential -Name "WAP Credentials" -Value (Get-Credential)Set-SmaCredential -Name "VMM Credentials" -Value (Get-Credential)#OR SET Credential Assets (by hardcoded plain text)Set-SmaCredential -Name "WAP Credentials" -Value (New-Object System.Management.Automation.PSCredential ("domain\username", (ConvertTo-SecureString "MyPassword" -AsPlainText -Force))) -WebServiceEndpoint $smaWSEndpointSet-SmaCredential -Name "VMM Credentials" -Value (New-Object System.Management.Automation.PSCredential ("domain\username", (ConvertTo-SecureString "MyPassword" -AsPlainText -Force))) -WebServiceEndpoint $smaWSEndpoint

Note This is just a simple example of how you could make the SMA Assets (Variables and Credentials) portable from one environment to the next. I have used generic values, so be sure to update the script before use. Once updated, this should be executed on a SMA server where the associated Runbooks will be (or are) imported (or at least somewhere the SMA Module is installed and the SMA WS is accessible).


Example Execution Results

The following are a couple screenshots of the output from the Automated Tenant Deprovisioning Process:

First Execution

Removed a Cloud Service, VM Network, and User Role

image

Second Execution

As expected as part of the process, results in a verification that the Subscription “No Longer Exists”

image


What Else?

The initiation of this process follows the same concept as my Tenant Provisioning Runbooks – WAP/SPF Action Events driven automation. I simply used a different Action Event (Subscription.Delete). There are plenty of other Action Events you could key off of, and initiate Runbooks from…in fact, you could use this same concept to perform a different set of actions (via Runbook) when a Subscription has been Suspended. Yes, there is a WAP/SPF Action Event for Subscription.Suspend/Activate – so just imagine the possibilities (Start/Stop/Store VMs, etc.).


TechNet Gallery Contribution and Download

The download ( Windows Azure Pack Tenant Deprovisioning Automation Toolkit.zip ) includes (11) files.

Each of the examples above is represented within this ZIP file.

Download the Windows Azure Pack Tenant Deprovisioning Automation Toolkit from TechNet Gallery here:

BC-DLButtonDark


Thanks for checking out my latest blog post! For more information, tips/tricks, and example solutions for Automation within System Center, Windows Azure Pack, Windows Azure, etc., be sure to check out the other blog posts from Building Clouds in the Automation Track!

enJOY!