OpsMgr 2007: Performance and scaling issues with the converted Exchange 2007 management pack
When you import the converted Microsoft Exchange Server 2007 Management Pack for System Center Operations Manager 2007, a variety of performance and scaling issues may occur, including some or all of the following:
1. MonitoringHost.exe may consume high CPU on monitored Exchange servers.
2. Health Service send queues may fill up on monitored Exchange servers. When this occurs, agents may log an event that is similar to the following:
Event Type: Warning
Event Source: HealthService
Event Category: Health Service
Event ID: 2023
Date: <date>
Time: <time>
User: N/A
Computer: <computer_name>
Description:
The health service has removed some items from the send queue for management group
"<MG_name>" since it exceeded the maximum allowed size of 15 megabytes.
3. Management servers may log HealthService 2115 events for either or both of the following workflows:
- Microsoft.SystemCenter.DataWarehouse.CollectEventData
- Microsoft.SystemCenter.CollectDiscoveryData
Example:
Event Type: Warning
Event Source: HealthService
Event Category: None
Event ID: 2115
Date: <date>
Time: <time>
User: N/A
Computer: <management_server>
Description:
A Bind Data Source in Management Group <MG_name> has posted items to the workflow, but has not received a response in 61 seconds. This indicates a performance or functional problem with the workflow.
Workflow Id : Microsoft.SystemCenter.CollectDiscoveryData
Instance : <management_server.fqdn>
Instance Id : {GUID}
4. The LocalizedText table in the OperationsManager database may grow to millions of records.
5. The dimension tables for the Event dataset (EventPublisher, EventLevel, EventCategory, etc.) may grow to millions of rows in the OperationsManagerDW database.
Cause
The diagnostic cmdlet script from the converted Exchange 2007 MP runs frequently and generates a large number of events. You cannot adjust the intervals for the rules that run the diagnostic cmdlet script and you cannot control the number of events it generates.
Also, the MP contains StateAlertDiscovery workflows that generate discovery data when monitor states are set.
Additionally, if you have clustered mailbox servers, rules may be targeted at physical nodes and at cluster computers. This causes two or three instances of the same rules to run on agents.
Finally, known issues in Health Service modules, MSXML 6.0 and Windows Script 5.6 may cause memory leaks and CPU spikes on agents.
Resolution
- Make sure that agents have Mommodules.dll version 6.0.6278.36 or higher. You can find the latest version in KB959865.
- Make sure that agents running Windows Server 2003 have Windows Script 5.7 installed (https://support.microsoft.com/kb/955360).
- Make sure that agents have MSXML 6.0 hotfix 968967 installed (https://support.microsoft.com/kb/968967).
- Run the script from the TechNet topic to clean up localizedtext and publishmessage tables: https://technet.microsoft.com/en-us/library/dd789073.aspx.
- Deploy hotfix 969130 (https://support.microsoft.com/kb/969130) to fix a known issue with Event grooming in the OperationsManagerDW database.
- To groom Event dimension tables, set up a SQL job to run the following script:
========
USE OperationsManagerDW
-- groom dynamic dimentions
DECLARE @MaxDataAgeDays int, @DatasetId uniqueidentifier
SET @DatasetId = (SELECT DatasetId
FROM dbo.StandardDataset
WHERE SchemaName = 'Event')
SELECT @MaxDataAgeDays = MAX(MaxDataAgeDays)
FROM StandardDatasetAggregation
WHERE (DatasetId = @DatasetID)
DELETE e
FROM EventCategory e
WHERE LastReceivedDateTime < DATEADD(day, -@MaxDataAgeDays, GETUTCDATE())
OPTION (RECOMPILE)
DELETE e
FROM EventChannel e
WHERE LastReceivedDateTime < DATEADD(day, -@MaxDataAgeDays, GETUTCDATE())
OPTION (RECOMPILE)
DELETE e
FROM EventLoggingComputer e
WHERE LastReceivedDateTime < DATEADD(day, -@MaxDataAgeDays, GETUTCDATE())
OPTION (RECOMPILE)
DELETE e
FROM EventPublisher e
WHERE LastReceivedDateTime < DATEADD(day, -@MaxDataAgeDays, GETUTCDATE())
OPTION (RECOMPILE)
DELETE e
FROM EventUserName e
WHERE LastReceivedDateTime < DATEADD(day, -@MaxDataAgeDays, GETUTCDATE())
OPTION (RECOMPILE)
========
Note: Simply save this as a SQL file to use it. A copy of the resulting SQL file is included in the attached ZIP file below.
- If you have clustered mailbox servers, the following management pack contains a number of helpful discovery and rule overrides to reduce redundant workflows:
Link to Exchange.Cluster.Overrides zip file download
This MP requires the Microsoft Exchange Server 2007 MP (version 6.0.6461.0) and the Windows Server Internet Information Services MP (version 6.0.6539.0).
Once you import the MP, populate the following groups:
- Exchange 2007 CCR Cluster Names
- Exchange 2007 CCR Clustered Mailbox Servers
- Exchange 2007 CCR Physical Nodes
After these groups are populated, open the OpsMgr Command Shell and run the following cmdlet:
Remove-DisabledMonitoringObject
This ensures that the cluster computers (the network name resources from the default cluster group) will not be members of the following classes:
Microsoft Exchange 2007 All Servers Installation
Microsoft Exchange 2007 Mailbox Servers Installation
Microsoft Exchange 2007 Mailbox - Physical Computers Installation
Microsoft Exchange 2007 Mailbox - CMS and Physical Computers Installation
Ex. Common
Ex. Mailbox
It ensures that the clustered mailbox servers will not be members of the following class:
Microsoft Exchange 2007 Mailbox - Physical Computers Installation
It ensures that the physical nodes in the clusters will not be members of the following classes:
Microsoft Exchange 2007 Mailbox Servers Installation
Ex. Common
Ex. Mailbox
It also disables the Test-ServiceHealth and Test-SystemHealth diagnostic cmdlets on passive cluster nodes.
Finally, it prevents cluster computers from being discovered and monitored as IIS servers. This means that IIS components will only be monitored on physical computers.
Hope this helps,
Michael Sadoff | Senior Support Escalation Engineer