Where does the time go? -519 Jet_errLogSequenceEnd


Enterprise Communications Support (formerly known as Exchange Messaging Support) has recently seen an increase in support incidents around the -519 Jet_errLogSequenceEnd issue. In this blog I will explain this issue in detail, why it may occur, how to recognize it, and what to do next.

Exchange has had a robust data engine that provides an ACID database from the early days; the transaction-based JET database engine ensures all database changes are Atomic, Consistent, Isolated and Durable. We accomplish this with the use of transaction log files. In Exchange Server 2000 and 2003, we introduced the concept of Storage Groups. Storage Groups can contain up to 5 databases, but share one set of common Transaction Logs. To differentiate between Storage Groups, all Transaction Log file names contain a prefix, followed by a 5 digit hexadecimal number, like so:

Enn00001.log

Where nn is the number of the Storage Group, typically 01, 02, and so on. As transactions (changes such as new e-mail, mail being read, tasks created, views built, etc) occur Exchange records them in sequentially numbered logs. This allows us to recover from certain database problems by knowing exactly which log comes next in a replay sequence. In Exchange 2000 and 2003, the transaction log file name length is 8 characters long. Since 3 of the characters form the file name prefix, 5 remain. Thus, the largest possible hexadecimal number we can represent is

EnnFFFFF.log

The actual largest number ESE in 2000/2003 will allow is FFFF0. That number in decimal is 1,048,560, representing the maximum number of log files we can write sequentially before running out.

With over one million 5 MB log files available, in some cases, it can take as long as a few years to hit this condition. When the transaction log sequence is exhausted, the Microsoft Jet database engine returns error -519, JET_errLogSequenceEnd to the Information Store. Depending on which version of Exchange 2000 or 2003 you are on, this error will result in slightly different symptoms. These symptoms are described in the following Knowledge Base articles:

830408 Store databases are dismounted without warning or users cannot log on to their mailboxes in Exchange Server 2003 or in Exchange 2000 Server

http://support.microsoft.com/default.aspx?scid=kb;EN-US;830408

896001 An event is not logged in the Application log before the last available transaction log in the sequence is used in Exchange 2000 Server

http://support.microsoft.com/default.aspx?scid=kb;EN-US;896001

In the latest revisions for Exchange 2000 and 2003, we added some fixes to warn you of this impending problem and prevent it from occurring. Store will warn you when you are nearing the end of the log sequence with the following event:

Event Type: Warning
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 514
Description: Information Store (2748) SG2: Log sequence numbers for this instance have almost been completely consumed. To begin renumbering from generation 1, the instance must be shutdown cleanly and all log files must be deleted. Backups will be invalidated.

If you have databases in Storage Groups that have been online contiguously for several years, you should be monitoring your transaction log sequence and the Application Log for this event. When you see this, it’s time to reset the transaction log sequence.

Notice that if you miss the ESE 514 warning your databases will dismount and generate the following events:

Event ID: 1159
Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Description: Database error 0xfffffdf9 occurred in function JTAB_BASE::EcEscrowUpdate while accessing the database “First Storage Group\Mailbox Store (SERVER)”.

Event ID: 9518
Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Description: Error 0xfffffddc starting Storage Group Path_of_Storage_Group on the Microsoft Exchange Information Store. Storage Group – Initialization of Jet failed.

OK, great, what the heck does 0xfffffddc mean? Glad you asked! You can look up Exchange error codes here: Microsoft Exchange Server Error Code Look-up

Note that error translates to Jet_errLogSequenceEndDatabasesConsistent:

Err 0xfffffddc –
# for hex 0xfffffddc / decimal -548
  JET_errLogSequenceEndDatabasesConsistent esent98.h
# /* databases have been recovered, but all possible log
# generations in the current sequence are used; delete all
# log files and the checkpoint file and backup the databases
# before continuing */

If you attempt to mount databases in this condition you will discover another cause for this common error in the Exchange System Manager:

An internal processing error has occurred. Try restarting the Exchange System Manager or the Microsoft Exchange Information Store service, or both.
ID no: c1041724
Exchange System Manager

Let’s Get Fixed!


Now that I’ve explained the issue and how it will be reported in the event logs, here’s how to correct this problem:

KB 830408 has a workaround section that describes how to reset transaction log sequence manually. However, we have included this functionality in the Microsoft Exchange Troubleshooting Assistant “Reset log generation number task”. We talk about that here:

MSExchangeIS 9518 (0xfffffddc): Exceeded the Maximum Number of Transaction Logs Available for this Storage Group

Microsoft recommends using the Troubleshooting Assistant (ExTRA) because it automates the process of verifying the health of your databases and resetting the transaction log sequence. Download the Microsoft Exchange Troubleshooting Assistant v1.0 and follow these instructions to reset transaction log sequence: http://technet.microsoft.com/en-us/library/aa998611(EXCHG.80).aspx

We also recommend monitoring all your Exchange 2000 and 2003 Storage Groups’ transaction log sequences to avoid a potential temporary outage of service. Monitor your application logs for the ESE 514 events.

Now, while it is theoretically possible to hit this condition in Exchange 2007, it will probably take a really long time. For Exchange Server 2007 we decreased the log file size to 1 MB but increased the transaction log file name length to 11 digits, meaning we can go up to

EnnFFFFFFFF.log

Due to the way ESE math works our upward limit is actually 7fffffec log files, but that is still a HUGE number. After figuring in the change in size to 1 MB, that’s still about 409 times the number of logs we could generate in 2000/2003. Read more about this improvement in the following TechNet Magazine article located here.

Corbin Meek
Enterprise Communications Support






Share this post :

















Comments (10)
  1. Karsten says:

    It seems the counting is starting with 00 for Logfiles of SG1 in our case (Exchange 2003 EP): E0095F9A.log.

    Thanks for this Info – an other Logfile Numbering is at: E01D7DFD.log. So I should reset this?

  2. Corbin Meek says:

    Hi Karsten,

    Your observation on the log file prefix is accurate, we start numbering storage groups at E00, not E01 as the blog may have implied.  Good catch!

    As far as resetting transaction log sequence for the SG you mention, E01D7DFD.log, that translates to 884221 decimal so you have some time before you run out.  You can wait until you see the ESE 514 event indicating it is time to reset your sequence, or reset at your next scheduled downtime, whatever is most convenient for your organization.

  3. Karsten says:

    Thanks for your reply.

  4. Will Mays says:

    Corbin,

    Am I correct in saying the the sequence is reset by an online full backup of the information store by an exchange-aware backup application?

    Will.

  5. Pradeep John says:

    Nice blog. Do Exchange 2007 server supports upto 4294967295 transactional log files?

    When Error -514 is triggerred? Is there is any limit?

  6. Pradeep,

    Exchange 2007 supports 2 billion log files (2147483628) which is 7fffffec in hexadecimal.

    Ross

  7. Pradeep John says:

    Thanks Ross. What is the log limit set in Exchange 2007 to trigger Error -514?

  8. Exchange says:

    Pradeep/all

    We are planning to update this blog post to include some information about what triggers Error -514

    stay tuned!

  9. Corbin Meek says:

    Will: No.  An online backup by an Exchange aware backup application using the Exchange backup API does not reset log file sequence.  Only the steps outlined in the supporting documentation will do that.

  10. austinmc says:

    Nigel Tufnel: The logs all go to eleven. Look, right across the board, eleven, eleven, eleven and…

    Marty DiBergi: Oh, I see. And most logs go up to five?

    Nigel Tufnel: Exactly.

    Marty DiBergi: Does that mean it’s louder? Is it any louder?

    Nigel Tufnel: Well, it’s 6 louder, isn’t it? It’s not five. You see, most blokes, you know, will be playing at five. You’re on five here, all the way up, all the way up, all the way up, you’re on five on your Exchange 2003. Where can you go from there? Where?

    Marty DiBergi: I don’t know.

    Nigel Tufnel: Nowhere. Exactly. What we do is, if we need that extra push over the cliff, you know what we do?

    Marty DiBergi: Put it up to eleven.

    Nigel Tufnel: Eleven. Exactly. Six louder.

    Marty DiBergi: Why don’t you just make Five louder and make five be the top number and make that a little louder?

    Nigel Tufnel: [pause] These go to eleven.

Comments are closed.