Not enough storage is available to process this command.

  • Article

Today I am going to share with you a real story of troubleshooting desktop heap depletion issue.

One day a customer opened a case and said that an important application named xyzService.exe in their production environment ran out of storage. I joined the remote session and observed that when we ran some query command in this software, we received error message "Not enough storage is available to process this command". Learning from customer, if we restart this application, the issue will be resolved, but after a period of time, like 2 months, the issue will be back again.

Because this looked like memory leak issue, so we generated a complete memory dump. I checked the memory dump with !vm command, I saw that there were 16G physical memory in this system, and the available memory is 800M. The available memory is small, but it should not lead to such fault. I went through all the well-known resources like paged/non-paged pool, handles, committed memory etc., unfortunately I did not get any useful information. Finally I went to the desktop heap.

To check the desktop heap usage we can use !dskheap command. In the output I found that desktop heap for window station Service-0x6-453186fc$ exhausted.

------------------------------------------------------------------------------------

Winstation @fffffa80xxxxxxxx: (Service-0x6-453186fc$) SessionID:  0

Desktop    @fffffa80xxxxxxxx: (Default)

Desktop Heap Size:         786432 (0x   c0000) Bytes

Committed:                     782336 (0x   bf000) Bytes

UnCommitted:                  4096 (0x    1000) Bytes

Allocated:                        782000 (0x   beeb0) Bytes

Total Freed                         336 (0x     150) Bytes

Unused:                             4432 (0x    1150) Bytes

Used Rate:                                   (        99)%

------------------------------------------------------------------------------------

In session 0, each user account will be mapped to a unique window station, Service-0x6-453186fc$ is not a build-in window station, like Service-0x0-3e7$ is well-known for LocalSystem, so it is used for some user account created by customer himself.

Checked the window station object fffffa80xxxxxxxx in the handle tables of all processes (with !handle to dump handles opened in every processes), figured out that the process that referred to (had opened handle to) this object is an application named xyzUIApplication.exe. The account used to open this application was DOAMIN\xyzServiceAccount.

After talked with customer, I figured out that the whole story here is that, customer ran xyzService.exe with domain account DOMAIN\xyzServiceAccount in session 0, and xyzService.exe forks child processes xyzUIApplication.exe somehow periodically. Unfortunately the UI application did not exit as usually. This operation leaked the desktop heap in this specific window station Service-0x6-453186fc$ gradually until the depletion was detected by xyzService.exe while running queries.

To solve this issue, we simple configured the xyzService.exe and made it not forks UI application in session 0.

-Justin