Event ID 81 and FCS

Are you occasionally seeing this in the Application log in Event Viewer on your FCS server?

Event ID: 81
The execution of the following DTS Package failed:
Error Source: Microsoft Data Transformation Services (DTS) Package
Error Description: Execution was canceled by user

The Problem:

This event can be caused by one of two problems:

  1. On Windows Server 2008, the configuration of the DTS job may be missing a switch. This switch, /MaxConn: , controls how many DTS steps can run in parallel. Without the switch, the default number is used, which is 8 steps. By specifying the minimum value of 5, there is a fewer number of parallel steps, with less of a chance they will conflict with one another.
  2. A thread in the DTS job is running into contention with another thread that hasn’t finished writing to the same table due to disk write performance. In this cause, you essentially have too many FCS clients and too little hardware to handle them. The physical disks for the SystemCenterReporting database are too few, or you are using the same disks for both the SystemCenterReporting and OnePoint databases.

The Fix:

For issue #1, the /MaxConn: switch, you can change the configuration of the scheduled task that runs the DTS job to contain the switch, as follows:

"<installpath>\MOM.Datawarehousing.DTSPackageGenerator.exe" /silent /srcserver:<server> /srcdb:OnePoint /dwserver: <server>/dwdb:SystemCenterReporting /product:"Microsoft Operations Manager" /maxconn:5

Note: Before you save changes to the task, ensure that the text in the Program/script text box in double quotes (“). Saving without the double quotes will result in a corrupted task.

If after you implement the preceding fix you still see Event ID 81, you are likely also experiencing a performance issue, and you should move the OnePoint database and/or the SystemCenterReporting database to reside on separate physical disks.

Note: If the DTS job has failed for multiple days, you may have outstanding data in OnePoint waiting to be transferred to SystemCenterReporting. If this is the case, see Microsoft Knowledge Base article 899158 for information about how to configure the DTS job to transfer data in smaller chunks.

OnePoint is fine on a single mirrored set of disks. However, SystemCenterReporting needs multiple spindles; see this article for guidance:

https://blogs.technet.com/fcsnerds/archive/2008/09/25/fcs-with-mom-2005-database-guidance.aspx

If you have more than 5000 clients reporting to your FCS server, the SQL Server that hosts the SystemCenterReporting database should be SQL Server 2005 Enterprise Edition and have 8GB of RAM available for SQL.

The Workaround (until you can get more disks for the databases):

If the scheduled task DTS job is failing intermittently (once every few days), you can change the schedule of the DTS task to run more than once a day. Each time it runs, it picks up where it left the time before. Usually running 2 times a day will guarantee that it succeeds at least one time a day.

To change the scheduled tasks on Windows Server 2003:

  1. Open the Control Panel, and start Scheduled Tasks.
  2. Right-click on SystemCenterDTSPackageTask and chose Properties.
  3. On the Schedule tab, click to turn on the checkbox for Show Multiple Schedules.
  4. Click the New button and set it to run 12 hours apart of the previous tasks (1pm).
  5. Click OK, and enter the DAS account credentials when prompted.

To change the scheduled tasks on Windows Server 2008:

  1. In Server Manager, expand Configuration\Task Scheduler and then click Task Scheduler Library.
  2. You should see the SystemCenterDTSPackageTask.
  3. Right-click the task and click Properties.
  4. On the Triggers tab, click the New button.
  5. Create a new schedule 12 hours apart from the previous one (1pm).
  6. Click OK, and enter the DAS account credentials when prompted.