Exchange Online: Looping through large numbers of objects, a PowerShell example
Sometimes it is necessary to run a command against a large number of objects. The simplest way to do this is with a pipe like in this example where litigation hold is enabled for all users:
get-mailbox -resultsize unlimited | set-mailbox -litigationholdenabled $true -litigationholdduration 3660
However, as you get into thousands or tens of thousands of users this may start to run into a couple of problems:
- running time can become quite high. This is because the Pipe operation takes place in the memory of your local PowerShell client. All the data from get-mailbox has to be brought down to your PC before the Pipe operation can take place.
- high memory consumption (because all that data has to be stored before the pipe is executed)
- Disconnects of the PowerShell session
- You could get throttled while running the command
You can get around these problems with Matt Byrd's EHLO blog. It is posted here:
I have had some of my customers complain that this is too complex or state that they wanted to make customizations.
This blog is about a simpler version. I am attaching an example that loops through all the objects. As it does so it saves a record of every operation it completes. That way if the script fails or gets interrupted it can exclude the completed entries from what it has already done. The script regularly tears down the PowerShell session and recreates it.
Hopefully I have included enough comments in the example that you will be able to modify it to suite your needs. It is expected all the PowerShell code included here would be saved as a PS1 file and then modified to suit your needs. This particular example retrieves all User Mailboxes and then makes sure the RetainDeletedItemsFor property is set to the default of 14 days.
Thanks
Chris
The example:
#
# Change RetainDeletedItemsFor to a specific value for all usermailboxes in the Exchange Online
# tenant.
#
# Data for each mailbox is written to disk as it is processed. If the script is run twice
#with the same file name and path the script will load the file and use the file to exclude
#mailboxes that have already been processed.
# As each mailbox is processed a line is appended to the Output file. If the script is interrupted
#it can resume based upon the content of the file. Simply make sure that you specify the exact same
#path and file name the next time you run the script.
#
# Parameters:
# Script takes the absolute path to a CSV file. For example:
# c:\o365\monthlyscript\Dec2015LastLogon.csv
#
# Script author: Chris Pollitt
# Last Modified: 2017-12-07
# Script provided as is without warranty of any kind. Use at your own risk.
#
param (
[string]$FileAndPath
)
# Pull the full list of mailboxes. Only pulling user mailboxes since Shared don't normally have licenses.
# Using invoke command so that pulling the full list of mailboxes takes only 2-4 minutes. Without the invoke this would be
# much slower, consume more RAM and be prone to hitting throttling limits outside of the invoke-command.
# Requires PowerShell 4.0 or higher
[array]$Mbxs = @()
[array]$Mbxs = invoke-command -session $session -scriptblock {Get-mailbox -RecipientTypeDetails usermailbox -resultsize unlimited | select-object userprincipalname,retaindeleteditemsfor}
# NOTE the use of $Session in the statements used in the Invoke-command. The Invoke needs to use the variable
#tied to the current PowerShell logon. If you use a different variable please change this.
$OptionsForSession = New-PSSessionOption
$OptionsForSession.SkipRevocationCheck = $true
if( $Mbxs.count -eq 0 )
{
write-host "Failed to retrieve list of mailboxes. "
$Count = (get-mailbox -RecipientTypeDetails usermailbox -resultsize unlimited).count
write-host "Number of mailboxes in the Org is " $Count
write-host "If the invoke-command did not return this please let author know."
exit
}
Write-host "initial mailboxes found " $Mbxs.count
write-host "Elimintating mailboxes where RetainDeletedItemsFor is already the desired value"
$Mbxs = $Mbxs | ?{$_.retaindeleteditemsfor -ne "14.00:00:00"}
if($Mbxs.count -eq 0) {
write-host "All mailboxes are set to 14.00:00:00 for RetainDeletedItemsFor. Script exiting."
exit
}
else {
write-host $Mbxs.count "mailboxes are not set to 14.00:00:00 for RetainDeletedItemsFor. "
}
[array]$LastUserChanged = @() #declare an empty array
[array]$UniqueMbxs = @()
$Resuming = $false
if(test-path $FileAndPath)
{
write-host "Found the output file. Assume we are resuming from a previous run"
[array]$PreviousProgress = import-csv -path $FileAndPath
if( $PreviousProgress.count -ne 0) #Make sure file is not empty.
{
$Resuming = $true
[array]$UniqueMbxs = compare-object -ReferenceObject $Mbxs.userprincipalname -DifferenceObject $PreviousProgress.userprincipalname -passthru | ?{$_.sideindicator -eq “<=”}
#Assert that $UniqueMbxs is an array that only contains UPNs at this point. If $PreviousProgress had additional properties
#they are now stripped out because all we need is a unique way to identify the mailbox.
write-host "Previously " $PreviousProgress.count " mailboxes were processed."
write-host ($Mbxs.count - $PreviousProgress.count) "mailboxes remain."
If( $Mbxs.count -eq $PreviousProgress.count )
{
Write-host “Number of mailboxes equals number of mailboxes in source file. Exiting script.”
Exit
}
}
else
{
write-host "Attempted to import previous file. No records found in the file. Treating this as a new execution of the script."
[array]$UniqueMbxs = $Mbxs
}
}
else
{
[array]$UniqueMbxs = $Mbxs
}
$Mbxs = $NULL #Clear array not being used
$PreviousProgress = $NULL #Clear array not being used
#release memory of arrays set to $NULL
[gc]::collect()
write-host $UniqueMbxs.count
$loopProgress = 1
$StatsSession = get-pssession
$Credential = get-credential -message "Please supply the stats script with credentials"
write-host " "
foreach($m in $UniqueMbxs)
{
if( ($Session.state -ne "Opened") -or (($loopProgress % 100) -eq 0))
{
write-host " "
Write-host "Removing Existing PS Sessions"
# Destroy any outstanding PS Session
Get-PSSession | Remove-PSSession -Confirm:$false
# Sleep 30s to allow the sessions to tear down fully
Write-host "Sleeping 30s for Session Tear Down"
Start-Sleep -Seconds 30
Write-host "Creating new PS Session"
write-host " "
# Create the session
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" -Credential $Credential -Authentication Basic -AllowRedirection -SessionOption $OptionsForSession
$null = Import-PSSession $Session -AllowClobber
} #EndIf
if ($Resuming) #$UniqueMbxs can have two formats. This affects syntax used to reference the data
{
set-mailbox $m -RetainDeletedItemsFor 14.00:00:00 #NOTE there is an inherent assumption on this line that the set-mailbox
#is successful. There is no Trap or exception checking here. To make this more robust the command could be wrapped in a
#Try Catch structure.
$Last = $m
}
else # Script is not resuming after a previous run
{
set-mailbox $m.userprincipalname -RetainDeletedItemsFor 14.00:00:00
$Last = $m.userprincipalname #NOTE there is an inherent assumption on this line that the set-mailbox is successful. There is not Trap or exception checking here.
} #EndIf
write-host "." -nonewline #Print a period with every record processed. Gives visual indication script is running.
sleep 1 #This is 1 second, but might need to be adjusted upwards if the script hits throttling limits.
#with session teardown and recreate every 100 objects you might even be able to comment this out.
[array]$LastUserChanged += $Last
$Last | Export-csv -append -path $FileAndPath -notypeinformation #If file does not exist it should be created.
$Last = $NULL #Clearing $Last in the hope that forcing it clear after writing will prevent odd duplication issue on subsequent times through the loop
$loopProgress++
} #Next
#$LastUserChanged #dumps the contents of variable to the screen
write-host "Script complete."