ESE, SQL and your feedback!


As most of you know, Exchange servers utilize the Extensible Storage Engine (ESE) database. As you have told us loud and clear, Exchange rocks from a performance, scalability, and availability standpoint on ESE and we’ve worked hard to make it so.

However, for some time (it has been a topic of discussion since the development days of Exchange 2003) there have been questions and rumors around replacing the ESE database with SQL Server.

  • Q. Did the team consider using SQL Server for the next version of Exchange?
    Absolutely!

  • Q. Did it work and perform well?
    Yes! Some very smart engineers did amazing work, and we had mailboxes up and running using a SQL Server database.

  • Q. Did we ultimately decide to replace the ESE database with SQL Server for the next version of Exchange Server ?
    No.

Why not! you ask – It was ultimately determined that the best way to ensure we could drive compelling innovation into Exchange for 2010 and beyond was to remain committed to ESE.  Enhancements made to the ESE database for Exchange 2010 have yielded significant performance benefits, High Availability benefits and provided customers with further opportunities to lower their storage costs. As we march forward on a path of developing Exchange for both on-premises deployments and cloud services, there are a number of areas that you are looking to us to make further improvements and new feature investments in the product. After much debate about both the benefits and challenges in moving to SQL Server, the decision was made to stay on ESE at this time.

We continue to evaluate technology options at every step of our development process to deliver the best product and service to you and invite your comments.

The Exchange Team


Comments (39)
  1. Personally I’m happy that you stay with ESE. So I don’t have to deal with an additional product(SQL) just to run Exchange. Exchange is complex enough… :-) Christian

  2. I agree with Christian Schindler on this. Not only does throwing in SQL add additional overhead to the server, it throws in extra levels of complexity and cost to running Exchange.

    No offense, but Exchange can already be an expensive investment for small-to-mid-size businesses, so requiring SQL licensing would push it out of the cost-effective park for many administrators.

    You guys definitely made the right decision.

  3. Exchange Geek says:

    Great to hear you guys sticked back to the ESE in stead of embeding another product technology into it. But when you say you that  the decision was made to stay on ESE at this time, does that mean the future releases of the product will have the storage integrated with SQL?  As already said by others Exchange itself is a costly and complex solution for small size bussinesses and of course highly skilled men managing it too. In that case adding another product as a backend would’t be a nice idea. I am hoping the future releases adhering to ESE some enhanced and more scalable version of ESE instead of SQL or any other database technology. :-)

  4. AbhayK says:

    Gud Idea not going with SQL. it would have added more management n configuration overhead to the already existing complex structure. I hope MS continues with the ESE engine as its really gives excellent performance results.

  5. Mario Cenc says:

    I think replacing ESE with SQL DBs would significantly increase the complexity level of the product.

    Running high available and high-performance SQL servers is a full time job on its own…

  6. Jozsef Turza says:

    I like ese and the current path of exchange personally.

    Objectively if going with SQL we are able to better manage larger databases and this aids in perhaps archiving older data to another SQL database and keeping current information in smaller DBs we could get the same maybe better functionality than we have now.  Investment in storage infrastructure may be better.

    The downside is you lose flexibility in innovating with exchange because it is dependent on SQL progression.

  7. Filipp Geyer says:

    Hi,

    all the others are talking about more complexity & costs. I allways thought: if Exchange-Development is going to switch to MS SQL as engine, it would be highly integrated in Exchange. Meaning: you don’t install an additional MS SQL an Point your Exchange to it, neither do you have a MS SQL Managment Tool. From administrator’s Point of view, it will all be the same like now: You know, that Exchange uses a DB-Engine, but you can’t access it directly.

    So, Exchange-Team: What kind of SQL-Integration did you think about? A separate SQL-Server, which can be managed separate from Exchange, or an Engine, which is Managed all by Exchange-Tools like now?

    Bye

    Filipp

  8. Mihai says:

    To Filipp Geyer: It’s impossible to integrate exchange with the current SQL Server. You would have to modify the SQL’s design. It is a very bad ideea to modify SQL Server just to adapt it with a new Exchange. So, i only see 2 options: they integrate a SQL-like engine into Exchange, replacing the ESE engine or they make a new product, with separate licensing, "SQL Server for Exchange" :)

    Unfortunately Microsoft doesn’t seem to understand that there is a big market for Exchange in SMBs. Exchange is a complex product, which is good both for SMBs and for large companies; but it’s very expensive both to license, and to maintain.

    Of course, from the business point of view, i think they would choose the second option.

    To the Exchange Team: When are the E12 SP2 and E14 Beta 2 scheduled to be released?

  9. Hexchtor says:

    Since exch 2k3 development one guy from development team told me that sql engine is not optimal for exch db, because sql bd’s usually make continuos writes and random reads, and the ESE engine must work with random reads and writes.

    Apart from this is the COST, that will make MUCH complex an Exchange implementation, requiring not only Exchange specialyst, but also SQL…

  10. Matt says:

    This makes very interesting reading – and I am very glad you stayed with the ESE database engine. As others have posted, it’s just an added overhead which Mail Admins don’t need – AND it would give false impressions that the DBAs and Mail Admins in large corporates should trade database space…!

  11. Paul says:

    Mihai — Why are you saying "it’s impossible to integrate exchange with the current SQL Server"?  Statements like that drive me nuts when they’re not backed up with any facts.

    Personally, from an enterprise perspective, I’m disappointed that MS is again passing on SQL integration.  In larger environments, integrating with SQL would be substantially cheaper from a number of perspectives.  For one, we’d have a TRUE "single instance storage" — today, because of practical limits to how many mailboxes you can have in a mailbox store, we have a single message going to several stores on several Exchange servers.  That also means that other enterprise concerns (like legal discovery and litigation holds) will be a substantially easier topic to swallow when there is just a single database.

    For the bigger folks that already have an established SQL infrastructure and talented skillset, SQL is definitely the better way to go.  Personally, I’m disappointed that this is getting passed over yet again.  I’ve been waiting for this for years now.

  12. Aaron says:

    I’m also very glad you are sticking with ESE. And very happy with where you are going with it.

  13. Johan J. says:

    Ofcourse would another technology make things more complicated.

    However, i would imagine that the Sql server would come included with a default Exchange install. At the moment you do not spend that much time tweaking the ese database either.

    However, unlocking mailstores through a SQL-like interface would make managing the separate mailmessages a lot easier. And being able to split the database engine away from the Exchange interface would mot only bring real single storage, but would also require just one expertise of running a high-performance SQL server. That way the exchange admin could just focus on mailflow and configuring. (if it would still be interesting is another question)

  14. Greg says:

    I’m really disappointed in this decision. Exchange 2007 is extremely difficult to backup & my DBAs can’t even work out how to do it at all. So I’m in a position where my SQL dbs are backed up every day but my mission critical mail store doesn’t get backed up b/c my DBAs can’t work out how to back up ESE without taking the whole environment offline.

    Backing up mission critical data is SUPER, SUPER important & Microsoft imposes a stupid requirement that customers must maintain TWO sets of IT support skills simply to back up these two critical data stores!!!

  15. Jim says:

    Exchange difficult to backup? Just get a copy of a dedicated backup application like BackupExec and you can backup the mailstore easily … I use BackupExec 12 to backup Exchange 2007 (full daily backup) and the performance is quite high, no downtime at all. I don’t see why you would need to take Exchange offline to back it up, unless you are backing up the raw files (ouch!), which is totally unnecessary. If you don’t backup the database and flush the transaction logs then the transaction logs will start growing until you are out of free disk space!

    I am sorry but if your DBA’s are having trouble figuring this one out then I have doubts about their professional qualifications (no offense). This kind of information can be found in TechCenter, otherwise just search Google …

  16. Ian says:

    I agree with Paul.  The current database technology is an administrative nightmare in large organizations with low IT salary budgets.  To keep restore SLA’s low and keep database performance acceptable, my organization has over 30 databases for our 1400 users.  It does not have to be this way.

    To those who are concerned about complexity or cost, there is absolutely no reason that SQL server could not be the underlying DB technology for exchange with no changes to the licensing or GUI administration interfaces.  Yes, there would be a change in interacting with the database at the command-line and scripting level for backups and maintenance, but that’s a *good* thing.  As for cost, anyone suggesting that Exchange would “have to” get more expensive if it was running on a SQL database would need to explain why I wouldn’t get a refund for my ESE database.  The point of course being that software pricing is all funny money anyway, if MS wanted to do this they could.  

    But apparently they don’t.  The canned “it’s not the right solution right now, but we’ll continue to look at it” is the same thing MS said when 2007 and 2003 were released.

    Here’s a suggestion to MS:  If you really don’t want to put a decent DB backend on Exchange, at least let us run it on any DB that we choose.  Continue to supply ESE with Exchange, but let teh customer substitute any SQL database for storage.  That way you don’t have to upset your SQLcustomers by ‘giving away’ SQL server, but the rest of us can make the choice to run a different backend.  How about it?

  17. Juan S. says:

    I have to agree with Paul. SQL could extend Exchange features substantially in the world of discovery, compliance, single instancing, and high availability. These avenues are best handled by third party vendors, thus forced to use busy MAPI services to access and mine data.

  18. Jonathan M says:

    Count me in on being disappointed with this decision.  One of the biggest issues Microsoft faces is supporting so many different products and keeping them alive under the guise of complexity.  More accurately, its the cost of development that is probably the concerning factor.

    Nonetheless, Microsoft should be forward thinking and moving down a SQL path sooner rather than later.  Get rid of the products that don’t have a future.

    Sorry, my opinion is maintaining ESE is one more reason why I should skip Exchange 2010.

  19. Tristan says:

    Ditto for the disappointment. Yes ESE performance is good and getting better, but SQL performance is also strong if correctly specced and maintained. Surely a switch to SQL would allow Microsoft to make better use of its design teams to the benefit of all, and its customers would be better able to consolidate efforts.

  20. pesos says:

    Lots of uninformed comments here.  Obviously there are numerous benefits to moving to sql, and the cost issue would not be a factor; it would be built into the product.

    That being said, it’s quite understandable that for 2010 it was more important to spend development resources elsewhere, and the feature set of office/exchange/sharepoint 2010 is very exciting.

    Hope you guys have time to move to SQL for e15!

  21. bday says:

    Frank, you have to be kidding. There is no way Exchange would have the enterprise market cornered if it was that kind of hackjob application.

    I don’t see how going to SQL would be a "sure thing" benefit to Exchange. Sticking with ESE/Jet allows the Exchange team to continue crafting it in ways that benefit Exchange itself. Moving to SQL would mean you’d have to write Exchange to work with SQL instead of writing ESE/Jet to work with Exchange.

    The Exchange team would be relying on whatever improvements the SQL team made to SQL and hope it worked well for Exchange instead of making changes they *know* will work. I’m sure there’d be plenty of co-development, but I don’t see it as being a guarantee.

  22. Ramy Messiha says:

    I like ESE , and I do recommend to stay on this technology , we don’t want to deal with SQL, just focus on Exchange.

  23. Suspect says:

    I see all the Comments flying arround and none of witch has all the facts of why MS stayed with ESE. MS  Probaly did intensive testing to and must have had real good reasons for staying on ESE.Untill we have all the facts we all can agrue about this still we all blue in the face. I do feel that witch look lyke the general feeling of all the comments Sql DB would couse extra overhead the QA is how big of a overhead.

  24. John says:

    Silly me, I thought they released new versions to make money, but Frank is smarter than us.  Why do Honda and Toyota keep releasing new models?

  25. Laurent Miltgen-Delinchamp says:

    "it has been a topic of discussion since the development days of Exchange 2003": wrong, it has been a topic since at least 1998 and the early development stages of Exchange 2000. I was attending the MEC (Microsoft Exchange Convention) at Boston in 1998 and it was one of the most commented and hot topic! Almost 11 years talking about that!

  26. Richard says:

    Since ESE is essentially Jet, and Exchange is a 64-bit only product, when can we expect to see 64-bit Jet OLEDB drivers?

    Microsoft have previously stated that they will *never* provide 64-bit OLEDB drivers for Jet, and suggest, for example, using a 32-bit SQL Express instance to connect a 64-bit SQL instance to a Jet database or Excel spreadsheet. This is a ridiculous suggestion – especially as Access is still going to be sold as part of Office 2010. What’s the point in selling a product that can’t be used, unless it’s a cynical money-grabbing rip-off???

  27. Laurion Burchall says:

    ESE isn’t the Jet you are thinking of. This blog entry has a discussion of the confusion:

    http://oakleafblog.blogspot.com/2006/07/red-vs-blue-jet-database-api-confusion.html

  28. Raveendran Chinnasamy says:

    SQL integration with Exchange will move forward in right direction instead of keeping with ESE. I dont  understand  people  talks about complex with SQL whereas Exchange itself more complex than SQL . :-)

  29. Petri says:

    Why you even published this "news"?

    1. You did not explained the reason to this decision. If you say the SQL does not performance well for Exchange, is this agreed with SQL team? How bad performance problems you found out? Where was the real information on this?

    2. "High Availability benefits", do you mean by that the SQL have not HA benefits? Really?

    3. "provided customers with further opportunities to lower their storage costs", I want cheaper solutions for the archiving. And for the cheaper storage solutions, try this:

      a) Today I do have so many DBs that SIS doesn’t give any benefits

          –> I need more storage

      b) Today I have started to use CCR

          –> I need more storage

      c) Users needs larger mailboxes, for backup/restore SLAs I need more

          DBs

          –> I need more storage

      d) E2010 and database groups…

          –> I need more storage

    4. I assume that one of the biggest reason (on the MS side and customers side) to not replace the ESE by SQL is worry about the personal work. If someone else do the database planning, maintenance and other works, we do have less work to do…

    5. Those who are saying that SQL and Exchange together are more complex than today, could you give some arguments for this? If your company does not have SQL knowledge, but Exchange, then I agree. But in all other cases I don’t see any reasons yet.

  30. CJ Phelps says:

    Exchange running on SQL back-end is a neat lab engineering idea, one I have thought of in the past, but like the majority of the posts, it would be a SysAdmin nightmare in an operational environment to configure, tune, manage, support, and recover.  The storage hardware alone would have to be optimally configured for ultra-quick, simultaneous database reads and writes, and SQL Server continually tuned by an experienced DBA to manage a huge number of indices.  In SQL best-practices, the recommendation would be to have an OLTP database for current emails (<7 days?) and an OLAP database for older emails, further compounding the complexity and administrative overhead.  Definitely stick w/ ESE!

  31. Joel Mansford says:

    I agree with @Petri, there’s no information here.  It would be interesting to know the specific reasons why the SQL DB Engine (and I assume we are talking about the SQL DB Engine rather than the MS SQL Product) is not a suitable replacement for ESE.

    I’m sure the Exchange team have good reasons to use what is now a specific-purpose storage engine (ESE) rather than a general purpose one (MS SQL).  

    To all the other posts regarding complexity, I would argue that there are more tools out there for the SQL Engine than there are for ESE.  Also it’s all down to how the SQL Engine is ‘hidden’ within Exchange.

  32. John says:

    Some of the arguments here are very disappointing. Why would someone even consider moving Exchange to an SQL Server back-end? It works fine on the current ESE database and I would leave it at that.

    Exchange in itself is by no means a complex product – in the same way Active Directory or indeed SQL Server is not complicated. Most people simply don’t have the experience in configuring it – which in itself makes things hard to understand and particularly complex. The Exchange team have made things much easier in Exchange 2007 (admittedly some things need improving – like all software written to stringent deadlines) and I have no doubt they will continue to improve matters in 2010 – the removal of storage groups is a bonus, for example. Trying to move the product to an SQL Server back-end will, in my opinion, simply add complexity to a product which many people don’t fully understand.

    I agree with the previous poster who went into detail about how the ESE engine is written for Exchange. If we moved to an SQL back-end, who would be coding the database software? The SQL Server team. Who would be making Exchange work with this database product? The Exchange team. How would this removal of the ability to make changes to the database structure impact the product? Lots. How much harder would development be? A lot. How will prices be affected? They’re go up.

    Don’t get me wrong – SQL Server is a great product with great uses. If you want to run database-integrated applications, great. But when you have a product as big as – if not bigger – than SQL Server which could potentially be put into an SQL Server database, it is too much dependency – more than I would like.

    The comment regarding ease of backup and restore from SQL Server has no basis to it. Like a previous poster our backup team have deployed Backup Exec 12.5 – and have NEVER had a problem with using GRT to recover data. It is quick and easy. However, an SQL Server DBA should not be charged with backing up Exchange data – either a trained backup admin or an Exchange admin should be doing so. Without any knowledge of Exchange it becomes very difficult to configure a good, working backup – and to restore if disaster strikes.

    I love Exchange and would never move away from it, but I would rather see development time go into adding new features – improving the functionality of the Management Console, for example, or improving clustering and co-existence with legacy systems without massive hardware/software investments being necessary (a big problem with 2010) – rather than trying to make an already excellent product work with an as-yet untested database platform.

  33. Jakub says:

    Hi all,

    personally I have been waiting anxiously for SQL integration for the same reason as some people here are talking about. And telling us that you will not integrate with SQL without any clear explanation is a shame really. I am always thinking what can go wrong with ESE database and in worst scenario – how can I recover data ? I believe with SQL and wide range of tools for it I would sleep better…

  34. Victor Tecson says:

    Being both an Exchange admin since version 5.5 and SQL DBA since SQL Server 6.5, I can bring up valid points for both sides of the argument (although I see more points towards SQL storage for those very large or public organizations that have to deal with terabytes of data and compliance issues).  I believe that this debate has been going on since the late 90’s as one of the people here had mentioned.  But I was more curious about the detailed findings of the Exchange Team, such as performance data etc.

    What disturbs me even further is the ambiguity about the future of Exchange’s database storage.  When consulting as an analyst, I have to sometimes give recommendations 5 years ahead or even more, which would affect many decisions such as budgets and purchases (relevant example is allocating more money to expand the SAN storage for Exchange versus piggy-backing off the existing SAN reserved just for SQL databases).  Then what happens when the Exchange team does a complete 180 ?!? (remember the IM chat/then no IM chat, collaboration portal/then no collaboration portal, M: drive/then no M: drive, public folders/then no public folders/then public folders again).

  35. David says:

    What’s really going on here seems to be an internal political battle for resources and a management desire to reduce development costs by designing, building and supporting one database engine for all Microsoft products.  

    It makes sense from their managements point of view and I can understand why Microsoft manage OUTSIDE of the Exchange Product Team keeps bringing this up.  (And they’re the only ones I hear internally talking about it as a ‘good’ idea.)

    It’s about control and specialization.  ESE is tweaked to support Exchange and AD is doing a superior job at it.  I manage 15,000 users across 130 stores and I don’t want more complexity.  Backups are easy and relatively fast.  I have a bunch HA options in 2007 and some very creative HA/DR options in 2010.  I want the Exchange Team to be in control of our database engine….  period!  

    I hope the Exchange Team continues to win the internal struggle.  OBTW, I’m not implying the SQL Team cares one way or another.  This is a management team cost issue not a custumer service issue to them.   ESE is still the customer service solution IMHO.

    Cheers

  36. Moray says:

    The main reasons I would like to see Exchange switch to a SQL backend are:

    1) reuse of a SQL server farm which is already designed for maximum throughput and reliability

    2) SQL’s relatively straightforward and successful tools for database repair

    3) assuming that the schema is made relatively straightforward, ease of developing applications which reach into the Exchange datastore. Current methods of programming against Exchange data are poorly understood and not easy to pick up.

  37. Ed says:

    I managed both an sql 2005/2000 and exchange 2003/2007 environment. I am very, VERY, glad that exchange and sql were not integrated. Some of the software that used sql 2005 didn’t work right after SP1 was installed. What would have happened if Exchange 2007 got an update that required the SQL server to have the latest updates. Presto, instant bomb. And if you think, well, have another SQL server/instance…Then whats the difference of that from keeping the ESE store? We also used BackupExec which worked great for backing up both SQL and Exchange. MAPI isn’t that hard to program against once you get the hang of it. Restores were crazy easy using BackupExec.

    I can see why using your current SQL Data Infrastructure for the exchange backend, but, when you add in the high load, and storage space requirements that an exchange store would require, wouldn’t that completely blow that out and you would then have to upgrade it? Being part of a very small (2) person I.T. group in a fairly large corporation, that should have had much more, I got a very large view of the overall picture, and all the details that fit in it. Also as a software developer, like I said, understaffed, I understand the difficulties in changing databackends from a system that was written specifically for one system, to another system, that is written to accomodate many systems. It would be almost a full re-write of everything except for the presentation layer.

  38. Erik says:

    Asside from the technical challenge, integrating SQL into Exchange would mean that the development of Exchange is forever tied to/dependent on SQL development.

    Could there be a small part of professional pride in that decision?

  39. Exchange_Geek says:

    All I can read in the entire thread is about added

    1) Management problems

    2) Administration problems

    3) More complexity

    Above three signifies the major headache if Exchange is integrated with SQL Back-End box.

    Tell me honestly, how many of us who were spoon fed with GUI based features got a major set back when we found powershell owning 80% of the operations on E12 (and further on E14). Didn’t we sit down and read and learn and ogt over the problem.

    To backup with facts E12 is a hot selling cake for Microsoft (I do not have access, else I would have given an estimate sales figure for E12).

    So, if there is improvement I think MS should think on terms of SQL. ESE/Jet has its own major disadvantages – not many of you realize that DB on Jet goes beyond corruption in many cases – Here Jet engine (along with PSS) will tell you SORRY and your DB is gone for good. Jet Engines has the problem with Named Property since ages.

    Some one above did mention – why did MS come up with such a announcement when it could not provide facts backing its news. It is something of sort tomorrow MS might release noather statement E14 is better than E12 – on what terms, on what basis – if not revealed whats the use of this announcement.

    (Off-the topic) recently NASA + ISRO guys announced water on Moon, didn’t they mention about the satellite, the mission, the date time (and other specifications etc). Cmon, MS you guys dont need to start more speculation about an on-going affair since past 5-10 yrs.

    Get rid of the Rumour once and for all with facts in front of the world.

Comments are closed.