“Squeaky Lobster” - Monitoring Version Store Counters

If you read nothing else in this blog post take a look at this https://msexchangeteam.com/archive/2006/06/15/427966.aspx .

Recently I was faced with a situation where a DC was receiving Event ID 623 events in the Directory Services eventlog. I had to quite a bit of searching for information on these events and came to the conclusion that AD guys don’t seem to have as many issues with the AD database as much as the Exchange guys. The reason I came to this conclusion, is that the majority of the information that relates to the JET database (or at least to the Version Store) that services both AD and Exchange is written in an Exchange context. Basically, the event relates to Version Store Exhaustion.

So before we go any further I thought I’d better refresh on what the Version Store is and found a couple of explanations. The official explanation is here https://technet.microsoft.com/en-us/library/cc772829.aspx , but I prefer simpler wording (being a simple guy) and think this article helps my understanding better “What is Version Store?” - https://msexchangeteam.com/archive/2006/04/19/425722.aspx ; yes I know it’s by Exchange guys, but that’s how this research generally panned out and it is still relevant. This is an extract and there is also a pretty good explanation of the event ID 623 in the rest of the article.

The Version Store keeps an in-memory list of modifications made to the database. This list has several uses:

1. Rollback - If a transaction needs to rollback it looks in the Version Store to get the list of operations it performed. By performing the inverse of all the operations the transaction can be rolled-back.

2. Write-conflict detection - If two different sessions try to modify the same record the Version Store will notice and reject the second modification.

3. Repeatable reads - When a session begins a transaction it always sees the same view of the database, even if other sessions modify the records it is looking at. When a session reads a record the Version Store is consulted to determine what version of the record the session should see.

4. Deferred before-image logging - A complicated optimization that lets us log less data than "other" database engines.

In simple terms, the Version Store is where transactions are held in memory until they can be written to disk. If something is preventing us from completing transaction or writing to disk we will consume this cache and the store will stop responding to request until there is room in the cache again.

By default the maximum amount of memory that can be allocated to the Version Store is the lesser of one fourth the amount of physical memory or 100Mb. This can be increased, but it is advised that this is a temporary solution to ease a specific issue. HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\”EDB max ver pages (increment over the minimum)” and it is a REG_DWORD. This value represents 16kbyte pages, e.g. a value of 1000 would be 1000 x 16kbytes.

Anyway, back to the reason for this article. I had my suspicions what was causing the 623 events (combination of an automated provisioning solution and\or Garbage Collection of ridiculous amounts of tombstones (another story) and underspec hardware; it was old tin). So, I decided that firstly, we needed to get some performance data, but the performance counters for monitoring the Version Store are not readily available and I needed to do some searching. Once again the majority of the information I found came from Exchange resources, but I found this https://technet.microsoft.com/en-us/library/cc961947.aspx which relates to Windows 2000, but is still valid. The results of the findings I will have to leave for another article.

To be able to view the allocated (16kb) pages of the version store you will need to carry out the following process:-

1. Copy the performance DLL (Esentprf.dll) located in SystemRoot\ System32 to any directory (for example, C:\Perf).

2. Run Regedit.exe, and make sure that the following registry subkeys exist:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESENT
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESENT\Performance
If these subkeys do not exist, you need to create them.
For more information about creating registry subkeys, see Windows 2000 Server Help.

3. Make sure that, under the Performance subkey, the registry values that have the following settings exist:

a. Open : data type REG_SZ : OpenPerformanceData

b. Collect : data type REG_SZ : CollectPerformanceData

c. Close : data type REG_SZ : ClosePerformanceData

d. Library : data type REG_SZ : c:\perf\esentprf.dll

e. Show Advanced Counters : data type REG_DWORD : 1 (This used to be called “Squeaky Lobster” and is pretty well known in Exchange circles; interesting blog on why it was called this from Brett Shirley here https://msexchangeteam.com/archive/2006/06/15/427966.aspx . It still works if you use “Squeaky Lobster” instead of “Show Advanced Counters” J).

4. Open a command prompt and change directory to SystemRoot \Winnt\System32 or to another folder that contains the files Esentperf.ini and Esentprf.hxx generated when Esentprf.dll was compiled.

5. (Optional) To verify that previous counter information is not present in the registry, at the command prompt, type unlodctr.exe ESENT .

6. To load the counter information into the registry, at the command prompt run Lodctr.exe Esentprf.ini.

To view the counters for the Database object, restart Performance Monitor. The counters are under Database and the one you need to monitor the Version store size is “Version Buckets Allocated”. This will show the number of 16kbyte pages allocated in the Version Store.