Summary: Microsoft Scripting Guy, Ed Wilson, talks about how to configure Windows PowerShell memory availability for specialized applications.
Hey, Scripting Guy! I really need your help. We are doing something that perhaps Windows PowerShell cannot do. At least, this is the way it seems. We have a huge file share, and we are parsing through it with a series of Select-String commands to find specific types of things. We are using Get-ChildItem to obtain files for us to parse, and we have filtered it out as much as is possible. The thing is, that when I say huge, I mean really, really huge. Absolutely GINORMOUS type of huge.
Anyway, we are perfectly resigned to the fact that Windows PowerShell will take some time to go through this parsing effort, and we have obviously tested everything on mock (much smaller) data prior to turning this thing loose. The problem is that Windows PowerShell runs for an hour or so, and then it stops with a System.OutOfMemoryException error message. It is bad enough that it happens, but it is horrible that it takes so long to occur. We make changes, reboot the server, wait for another hour, and boom!—it happens again. We have spent an entire week trying to make this work, and you are our last hope. I searched the Hey, Scripting Guy! blog, but I did not find anything helpful. So now’s your chance to be a real hero.
—AP
Hello AP,
Microsoft Scripting Guy, Ed Wilson, is here. Today is a great day. I got up, and fixed some nice English Breakfast tea with a bit of organic orange rind, some peppermint and spearmint leaves, a bit of crushed cinnamon stick, and a touch of lemon grass. I must say, it is a very refreshing cup of tea. Yesterday I had an awesome session with my mentee, Ashley McGlone. I am real proud of everything he has accomplished so far. So the week is going along perfectly. I am looking forward to this Thursday (August 1, 2013). We are having the Windows PowerShell User Group meeting in Charlotte, North Carolina. It will be awesome. With everything moving along smoothly, I thought I would take some time to try to catch up a bit with questions such as yours that are emailed to scripter@microsoft.com.
Configuring memory for Windows PowerShell
To configure memory resources for Windows PowerShell, you must launch Windows PowerShell with Admin rights. If you attempt to do anything on the WSMAN: drive as a normal user, you will receive the following “Access is denied” error message:
In addition to Admin rights, the WinRM service must be running. In Windows PowerShell 3.0 in Windows 8, this service starts on demand. Therefore, the first attempts to access the WinRM drive will result in a prompt to start the WinRM service. I use the Get-Service cmdlet to ensure that everything started properly:
PS C:\> get-service *win*
Status Name DisplayName
------ ---- -----------
Running WinDefend Windows Defender Service
Running WinHttpAutoProx... WinHTTP Web Proxy Auto-Discovery Se..
Running Winmgmt Windows Management Instrumentation
Running WinRM Windows Remote Management (WS-Manag..
Check and set the machine-wide setting
The first thing to do is to check and set the machine-wide memory setting. To do this, I navigate to WsMan:\Localhost\Shell in my Windows PowerShell console. I then use the Get-ChildItem cmdlet (dir is alias) to see my settings for everything. This is shown here:
PS C:\> sl WSMan:\localhost\Shell
PS WSMan:\localhost\Shell> dir
WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Shell
Type Name SourceOfValue Value
---- ---- ------------- -----
System.String AllowRemoteShellAccess true
System.String IdleTimeout 7200000
System.String MaxConcurrentUsers 10
System.String MaxShellRunTime 2147483647
System.String MaxProcessesPerShell 25
System.String MaxMemoryPerShellMB 1024
System.String MaxShellsPerUser 30
Set MaxMemoryPerShellMB
To make the change, I use the Set-Item cmdlet and change the value of MaxMemoryPerShellMB from 1 GB to 2 GB. This technique is shown here:
Set-Item .\MaxMemoryPerShellMB 2048
Now I use the Up arrow and change Get-Item to Set-Item. This command and its output are shown here:
Note I am already in WsMan:\LocalHost\Shell when I run the Set-Item command. If you do not want to navigate to the folder first, you can use this command:
Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048
I notice that a warning appears that states I also need to change memory settings for plug-ins. (This is true in Windows PowerShell 3.0.) Therefore, I navigate to the plug-ins directory to make those changes. But before I make any changes, I notice there are several plug-ins listed:
PS WSMan:\localhost\Plugin> dir
WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Plugin
Type Keys Name
---- ---- ----
Container {Name=Event Forwarding Plugin} Event Forwarding Plugin
Container {Name=microsoft.powershell} microsoft.powershell
Container {Name=microsoft.powershell.workf... microsoft.powershell.workflow
Container {Name=microsoft.powershell32} microsoft.powershell32
Container {Name=microsoft.windows.serverma... microsoft.windows.servermanag...
Container {Name=WMI Provider} WMI Provider
The thing that is confusing is that I need to make a memory change for each plug-in endpoint configuration that I target from the client. Luckily, I happen to know that the default Windows PowerShell endpoint is Microsoft.PowerShell, and that is the only one I need to change. I type the following command:
Set-Item .\microsoft.powershell\Quotas\MaxConcurrentCommandsPerShell 2048
The command results in a warning that states I need to restart WinRM and that the value for the plug-in will only work if it is less than or equal to the value for the global memory setting. Here is the command and the output:
Note I was in the Wsman:\LocalHost\Plugin directory when I ran the command to set the memory for the plug-in. If you do not want to navigate to the location, use this command instead:
Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB 2048
I use the Get-Item cmdlet to ensure that the new value took. Here is the command I use:
PS WSMan:\localhost\Plugin> get-Item .\microsoft.powershell\Quotas\MaxMemoryPerShellMB
WSManConfig:
Microsoft.WSMan.Management\WSMan::localhost\Plugin\microsoft.powershell\Quotas
Type Name SourceOfValue Value
---- ---- ------------- -----
System.String MaxMemoryPerShellMB 2048
Restart the WinRM service
Now I need to restart the WinRM service. To do this, I use the Restart-Service cmdlet. The command is shown here:
Restart-Service winrm
Just for fun, I close the Windows PowerShell console, and then reopen it. I rerun my Get-Item commands to see if anything has reverted. As indicated in the image that follows, everything is groovy.
AP, that is all there is to using the WSMAN drive to configure Windows PowerShell memory. Join me tomorrow when I will talk about more cool stuff.
I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.
Ed Wilson, Microsoft Scripting Guy
How can I found out which quota is reached? If I run in a remote session something like this:
cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd /C cmd
is going to fail due to ‘not enough quota’ error. I tried increasing all the possible limits as you described here but it didn’t work.
@anonymous: I’m having the same huge memory leak problem with the quest ad cmdlets when running my scripts through PowerGui, but when I run them through the Active Roles Management Shell, memory consumption remains still at around 70 MB.
@jrv
Yes, you can change the setting in 2.0 but it doesn't appear to be used.
@Alex Brassington this technique does not work on PowerShell 2.0 (I do not believe).
@All So according to JRV it should work on PowerShell 2.0. Cool. Thanks JRV
@Alex – here are the WMF team blogs on this for WMF 2.0:
blogs.msdn.com/…/configuring-wsman-limits.aspx
Note that this setting only affects WSMan shells and not PowerShell itself. These settings are only for the remoting component of WSMan calls WinRM. They can also be configured via WinRM at an command prompt.
Does this work with PowerShell 2.0?
The default MaxMemoryPerShellMB is only 150 and i've seen it breeze past that a lot of times. I've tried changing it and it doesnt' appear to have any effect, i set the limit to 100MB and set it to Get-Childitems for my C drive and it hit 150MB +.
What changes should we be seeing or am I missing something?
@Jrv
That makes more sense, that property only affects sessions that have been initated as remote sessions, not as ones started locally.
@All
PS C:scripts> $psversiontable.PSversion
Major Minor Build Revision
—– —– —– ——–
2 0 -1 -1
PS C:scripts> Set-Item WSMan:localhostShellMaxMemoryPerShellMB 2048
PS C:scripts> dir WSMan:localhostShell
WSManConfig: Microsoft.WSMan.ManagementWSMan::localhostShell
Name Value Type
—- —– —-
AllowRemoteShellAccess true System.String
IdleTimeout 180000 System.String
MaxConcurrentUsers 5 System.String
MaxShellRunTime 2147483647 System.String
MaxProcessesPerShell 15 System.String
MaxMemoryPerShellMB 2048 System.String
MaxShellsPerUser 5 System.String
PS C:scripts>
@Alex – how did you prove it wasn't used. The setting sticks and reports the change. Same as in V3.
The limits in WMF 2.0 may be lower than in WMF 3.0.
If you are still running PowerShell V1 then this will not work.
I meant @Rich
The memory problem I have with PS 3.0 is that it just consumes memory until my machine grinds to halt. Something relatively simple like getting all users and their login times from AD will use 2Gb of memory. I tried fetching the same information from all DCs and PS just used up the whole of my machine's memory. I was piping the results to Export-CSV so as far as I can see there's no need for large memory use anyway.
I should have mentioned I'm using Quest AD cmdlets and it turns out they have a memory leak bug. At least they did in 2010 and I'm still having the problem today.
I was having memory limit issues running powershell remote commands that launch java programs on that remote host.
I found setting the MaxMemoryPerShellMB to zero, effectively removes the limit and allowed my program to run
Set-Item .MaxMemoryPerShellMB 0
Thank you very much. Now I can process my map data.
This line above:
Set-Item .microsoft.powershellQuotasMaxConcurrentCommandsPerShell 2048
Should read:
Set-Item .microsoft.powershellQuotasMaxMemoryPerShellMB 2048