Dealing with Memory Pressure problems in MOSS/WSS

In the last couple of weeks I have seen several issues related to memory pressure situations in MOSS or WSS application pools so I think it is a good idea to discuss some aspects of this type of problem in a separate article.


What is a memory pressure situation?

First of all: what do we understand under memory pressure? To answer this question we need to understand the basic memory management concept of the .NET framework.

Managed Objects allocated by the .NET framework are stored on a so called managed heap. When more memory is required than available on the managed heap the .NET framework allocates a new heap segment which by default has 64 MB in size. So in order to extend the heap it is required that a free contiguous 64 MB segment is available in the virtual address space. From time to time the garbage collector will run and compact the memory inside the managed heap to ensure that new 64 MB segments are only allocated if really required.

In theory this would mean that the memory can grow up to 2 GB per process on a 32-bit machine – and if you are writing your own windows or console application in C# then you can indeed use nearly this amount of memory!

But with ASP.NET it is different. The reason is not ASP.NET itself but the fact that ASP.NET is hosted inside an IIS w3wp.exe worker process.

This w3wp.exe process does not only contain the .NET framework but also DLLs from IIS like ISAPI extensions and ISAPI filters. That would also not be a problem if these DLLs would be loaded one after the other as an contiguous block in the virtual address space. In reality this doesn’t happen.

Each DLL has it’s own preferred load address inside the 2 GB address space which causes the available virtual memory to be split into multiple different pieces separated by the dlls loaded into memory. Often the distance between two DLLs is smaller than 64 MB – which means that this memory is not available for the managed heap. And even if the memory segment is bigger than 64 MB (e.g. 100 MB) it means that after allocating 64 MB segments the remaining memory will be smaller than 64 MB (e.g. 36 MB if we look at the 100 MB sample from before).

See here for details: http://blogs.msdn.com/tess/archive/2006/09/06/net-memory-usage-a-restaurant-analogy.aspx

If there is no additional contiguous memory segment of 64 MB can be found in the virtual address space we are talking about a memory pressure situation.

Usually this will occur between 800 and 1000 MB.

With other words: whenever an ASP.NET worker processes exceeds 800 MB it can become unstable and out of memory errors are likely to happen. In addition you can usually also see a performance impact on your site as a large amount of CPU time is now used by the garbage collector which has to run much more frequently.


Common Reasons for memory pressure

There are many different reasons that can lead to memory pressure situations:

Reason 1: Running a web application in debug mode.

If a web application in debug mode, then every single ASPX page is compiled into a separate DLL. That means that the memory fragementation is heavily increased.

=> Always configure your web applications to run in release mode

Reason 2: Managed objects holding references to COM components are not correctly released.

Special care has to be taken for all managed objects which hold references to unmanaged COM components as these unmanaged COM components can hold references to unmanaged resources like file handles but also allocated memory which is not under control of the .NET framework which means that the garbage collector cannot see and free up this memory if it is no longer used.

Usually the managed components holding references to unmanaged components implement a Dispose() or Close() method to explicitly release the unmanaged COM components including the resources allocated by these components.

In case that these methods are not called the unmanaged resources are not released and increase the overall memory consumption. Usually these managed objects will finally release the COM components in their finalizer method when the garbage collector adds them to the finalizer queue but that can already be to late as the finalizer will not run together with the garbage collector – means it might be that not enough memory for further allocations is available at the time when it is required.

In SharePoint we have one object type that holds references to unmanaged COM components: the SPRequest object. Each SPWeb and SPSite hold a reference to one of these SPRequest objects. So in all custom code using SPWeb or SPSite objects it is vital to correctly dispose these objects to free up no longer required memory resources.

See here for details: Best Practices: Using Disposable Windows SharePoint Services Objects

When using the publishing features there is one additional object that needs to be correctly closed: PublishingWeb. The reason here is that the PublishingWeb object itself holds a reference to the associated SPWeb object. In order to release the resources of the bound COM objects it is required that the PublishingWeb object releases the associated SPWeb object.

This can be done using the Close() method of the PublishingWeb object.

=> Always ensure to correctly dispose all SPWeb, SPSite and PublishingWeb as discussed in the above article.

Reason 3: Hosting multiple web applications in the same application pool

Each independent web application usually ships with it’s own unique dlls and has a specific usage pattern for allocated objects in memory. Hosting multiple web applicaitons in the same application pool causes all DLLs for all web applications to be loaded into the same worker process and also objects specific to this web application to be created. This increases the memory pressure.

=> If your application requires a large amount of memory ensure that it is running in a dedicated application pool.

Reason 4: memory hungry application code

In some situations the application code is written in a way that each single request hitting the server results in a huge memory consumption. E.g. if large arrays or dictionaries are being allocated.

But also some of the out-of-the-box components can require huge amount of memory when incorrectly configured. Some components that are often responsible for high memory situation and participate in memory pressure are navigation controls and site map providers. For each item in a navigation control that has to be retrieved through the sharepoint site map provider SPWeb and SPSite objects will have to be created. Although the controls ensure that the SPWeb and SPSite objects are correctly disposed before the request ends there are often several of these objects in parallel in memory while the control is being rendered. You might have seen warnings like the following in the ULS log when this happens:

Potentially excessive number of SPRequest objects (53) currently unreleased on thread 13

The number of objects (here 53) and the thread number (here 13) will vary. This warning indicates that there are controls on your pages which require many SPWeb and SPSite objects.

To ensure that no memory pressure occurs it is vital that the number of SPWeb and SPSite objects required by such a navigation control and site map provider is as small as possible. So you should ensure that you are configuring the navigation controls with minimum depth rather than having a flyout menue that shows your site structure 5 levels deep.

If your site design really requires such a navigation control you should better feed it from a static XML file which is (e.g.) generated once a day rather than from one of the SharePoint site map providers.

=> Ensure that your site logic releases allocated memory as quick as possible and only allocates as much memory as really required. Also ensure to configure the navigation controls to not enumerate big parts of the SharePoint database.

What solutions do we have for memory pressure?

In ASP.NET there are more or less 3 approaches for this:

1) Follow all the steps outlined above to ensure that your application only allocates the memory it really requires (this is the approach you always should follow first!)

2) Using the /3GB switch? – sorry this cannot be used with SharePoint!
Although this solution would allow ASP.NET to use around 1.8 GB of memory before running into memory pressure situation it cannot be used with SharePoint.
The following article explains why using the /3GB switch in SharePoint is not an option:
https://support.microsoft.com/default.aspx/kb/933560

3) Switching to 64-bit architecture
In 64-bit architecture the virtual address space is no longer limited to 2 GB. This also means that memory fragmentation as discussed in the beginning of this article will not have the negative effects as in 32-bit architecture and the 800 MB limitation for ASP.NET does no longer exist.

35 Comments


  1. I just a found another great post about memory management and performance issues on the SharePoint platform.

    Reply

  2. Excellent article. Required reading for all SharePointies. Dealing with Memory Pressure problems in MOSS

    Reply

  3. We consistently have memory issues with our large SharePoint implementation. These are great tips that I can now pass onto the developers and help bolster our SharePont Dev standards.

    Reply

  4. Overview Windows SharePoint Services (WSS 3.0) and Microsoft Office SharePoint Server (MOSS 2007) have

    Reply

  5. Overview Windows SharePoint Services (WSS 3.0) and Microsoft Office SharePoint Server (MOSS 2007) have

    Reply

  6. A couple of weeks ago I wrote an article which explains how to deal with memory pressure situations in

    Reply

  7. One of the areas covered during the training I do on WSS Development is how to correctly dispose of objects

    Reply

  8. Un post qui vient un poil en doublon de ceux qui relaye 2 nouveaux articles de blog mais ces posts sont

    Reply

  9. In an earlier article I have discussed that all SPSite and SPWeb (and potentially also PublishingWeb)

    Reply

  10. Hi Stefan,

    This post displayes the information on what can cause a memory leak in the w3wp.exe process, is there anything that can cause a memory leak (under the context of MOSS server) in the SQL Server process: sqlservr.exe?

    I am performing a lot of imports, one after another, and I suspect that such thing may be the cause.

    Reply

  11. Hi Mor,

    import operations can lead to high memory usage on SQL server as well. That’s why we recommend 64-bit architecture on SQL server.

    But it should not lead to a leak.

    If there would be a leak it would be a bug in SQL server.

    Cheers,

    Stefan

    Reply

  12. Stefan Gossner has published a great article on the subject: In an earlier article I have discussed that

    Reply

  13. Might as well dive right in and make the inaugural posting useful… Anyone who's done some development

    Reply

  14. Performance is perspective that all the developers forget during development and it pops up and the end

    Reply

  15. I have discussed problems with missing dispose for SPWeb and SPSite objects earlier on my blog (e.g.

    Reply

  16. Stefan,

    Would it be possible to get your sharepoint w3wp to use more memory if you had a windows 2003 enterprise server (8 gig ram) with PAE enabled?

    Reply

  17. Hi Matt,

    no this will not help. PAE allows a computer to address more physical memory. Not more virtual memory.

    The virtual memory of a process is always 4GB. 2 GB system and 2 GB user mode RAM.

    So PAE would allow you to run more processes without a need of paging. But not to address more memory within a single process.

    Cheers,

    Stefan

    Reply

  18. Hi Matt,

    just to add: the solutkion for your problem is 64 bit technology.

    Then the virtual memory in a process becomes bigger.

    Cheers,

    Stefan

    Reply

  19. Thanks Stefan!

    I had a feeling that was going to be the answer.  Unless we can get our WFEs replaced with 64 bit servers, all we can do is add some more regular 32 bit WFEs to the farm.  Well that and write better code.

    Reply

  20. what are some debugging tools you use to identify such problems

    Reply

  21. I use WinDBG and SOS from Debugging Tools for Windows

    Reply

  22. There are several articles around discussing the dispose of SPWeb and SPSite objects, e.g.: SharePoint

    Reply

  23. Linki, które posłużyły mi przy tworzeniu prezentacji, z których czerpałem wiedzę, nakładałem ją na to

    Reply

  24. SPSite and SPWeb implement the IDisposable interface

    Reply

  25. I’ve been working on supporting and enhancing a client’s custom SharePoint solution for a while now.

    Reply

  26. I have dispose and close all the SP objects and also increase the virtual memory of application pool.

    but still got "out of memory" error while deleting the list items.

    Thanks in advance.

    Reply

  27. Hi Ganesh,

    I would suggest to open a support case with Microsoft if you need assistance to analyze what is going on.

    Cheers,

    Stefan

    Reply

  28. When do you add another WFE to the farm?

    Reply

  29. You have to do this when the average request execution time exceeds the frequency of incoming requests.

    Reply

  30. ..and how do you do that? Do you use PerfMon?

    Reply

  31. actually IIS logs are sufficient for this. Take the average time-taken – which is the request execution time – and compare it with the incoming request rate during busiest hours.

    Reply

  32. ok maybe i am missing something but how do i see it. do i have to setup any parameters in the IIS log monitoring service?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.