Outlook 2010 - network latency test results

Over the past 18 months I have been dealing with more and more cloud engagements.  One thing that crops up during the planning phase of all of these engagements is network link latency and bandwidth requirements.

Bandwidth prediction is something I am working on in the background, but network link latency seemed like a mystery!.  Everywhere I looked and everyone I spoke to had a different opinion on “maximum latency” for Outlook clients.

So.. I decided to try to devise a test that would let me model the impact of network link latency on Outlook client performance.  My lab already had a network emulator installed that was capable of simulating link latency so all that I needed was a script to automate Outlook to perform some common tasks, then record how long the script took to run – easy right? Smile

Automating Outlook…

I already had some scripts for Outlook 2010 from my bandwidth prediction work, so I decided to make use of them.  The scripts were written with AutoIT which is a regular part of my lab testing toolkit these days!

Software versions in use for all tests…

  • Windows 7 Service Pack 1 32-bit
  • Outlook 2010 Service Pack 1
  • Exchange Server 2010 Service Pack 1

Note: All tests began from a cold start i.e Outlook was not running.

Test definitions

Note: All clients were tested from 0ms added latency to 1000ms in steps of 100ms and then up to failure point from 1000ms onwards in steps of 500ms. Failure was deemed to be if any single test failed.

Test 1. Sending

  • Send 5 messages with 2 x 50kb attachments

Test 2. Opening

  • Open 10 messages with 2 x 50kb attachments
  • save attachments to disk

Test 3. Delegates

  • Open delegates
  • Add in a new delegate
  • Remove delegate

Test 4. Out of Office

  • Open out-of-office
  • Enabled message and set OOF
  • Switch OOF off

Test 5. Calendar

  • Open calendar
  • Add in shared calendar
  • close shared calendar

 

Notes:

  • For all cached mode tests a final sync was performed and test did not completed until the outbox was empty.
  • Outlook Anywhere was configured to connect over HTTPS on fast networks to reduce start-up delays.
  • Windows 2008 R2 CA PKI was used to issue Exchange certificates in the lab.

Results

I decided to run tests 1-5 sequentially for three types of client connections

  • Outlook 2010 Online
  • Outlook 2010 Cached Mode
  • Outlook 2010 Outlook Anywhere (RPC/HTTPS)

Note: All latency values represented are round-trip times RTT as reported by ping.exe from a command line.

Combined test case results

image

What is evident from this set of results is that the impact of network link latency is linearly related to outlook client performance.  Certainly when we consider a broad range of functions.  Cached mode dramatically reduces the user wait time for send/receive functions, but it was obvious watching the test run that whenever actions were requested that ran outside of the OST file, such as delegate access or out-of-office requests large delays were experienced at higher latency values.  I decided to set the maximum test run time limit at 200 seconds, which was double the test run time with no additional latency applied.  Client behaviour when the test duration was taking 200 seconds was the poorest performance level that I perceived to be acceptable by anyone for this test.  I judged that if any single operation took 10 seconds or more to display a dialog box or the client appeared to hang then it was unacceptable for the end user. 

In this combined test the limiting task was working with the delegate permissions, which was taking approximately 9 seconds to complete at the maximum test run time.  Accessing delegate permissions was similar amongst all connection modes. 

The results for Outlook Anywhere were confusing initially, although the majority of the difference between Cached and Cached-OA is due to the additional time it takes for an Outlook Anywhere connected client to establish its initial connection.  Typically an OA client will take around 10 seconds longer to establish a connection when compared to a normal RPC MAPI connection.  This rises to around 30 seconds at 800ms latency.  Under use both Outlook Anywhere and Cached mode performed with very similar results.

Combined test conclusion

The test results for me were quite interesting.  Prior to this testing I had assumed that Outlook tolerance for link latency was was much worse than the test results show.  This is no doubt due to changes in adaptive TCP windowing on Windows 7 plus improvements in Outlook 2010.  The bottom line is clear though, Outlook 2010 on Windows 7 is able to tolerate quite high network link latency values before performance begins to suffer noticeably.  In the case of Cached mode and Outlook Anywhere both clients were able to tolerate 500ms round-trip-time latency without problem and if your main tasks are sending and receiving mail only, cached mode actually provided a good experience at up to 1000ms round-trip-time latency.  Online mode was far more sensitive to network link latency as expected and performance became noticeably slower at 190ms round-trip-time and above.  Online mode was still usable above 200ms latency but felt sluggish.

2011-06-15_1310

Failures

One thing that I didn't account for during this test was that the clients would behave differently as latency increase.  My automated script initially didn't cope very well when Outlook in online mode was subjected to high latency values.  The long pauses before dialog boxes were displayed made things quite tricky at times, AutoIT is extremely capable though which really helped in this testing!.

The following failure points were noted..

  • Outlook 2010 Online Mode – General script failure at 800ms (gave in trying to make it work – “Outlook is not responding” )
  • Outlook 2010 RPC/HTTPS – Delegates page failed to open at 2000ms
  • Outlook 2010 Cached Mode – Delegates page failed to open at 2000ms

Note: Although the cached mode clients failed at 2000ms for delegate access the client continued to provide usable mail performance.  I plan to run further tests in cached mode to see where the break point is for mail only users since this represents the majority of users.  Keep watching my blog for more information on this test Smile

Other thoughts…

This test was designed to isolate the effects of network link latency on client performance, however the reality is that network link latency is only one part of the Outlook client performance experience.  In practice it is more likely that poor client performance is caused by local hardware limitations or insufficient resources allocated on the Exchange Server itself.   This testing does show however, that consolidating Exchange Server resources into central locations is viable with Windows 7 and Outlook 2010.

OK, that's great, but what about OST file resync?

So.. this is an interesting question that came to mind while I was performing this testing.  We can provide a good user experience in normal day to day tasks over quite latent network links, but what happens if I need to rebuild my OST file?  OST file resync is a fairly common occurrence; even if we just consider cases of SATA HDD failure (5% AFR typically), that means in an organisation of 30,000 laptop users there will be approximately 1500 OST resync’s per year (7.5 per working day!) due to HDD failure.  I was curious about how long it would take to synchronise an OST file under specific latency conditions, so I ran some tests….

I created a 100MB mailbox then timed how long that took to synchronise down as latency increased, I then calculated how many seconds it took per 1MB of data.  My plan was to use this to extrapolate how long larger mailboxes would take to synchronise down.

Note: I had initially planned on doing this experimentally but then the reality of having to run 10 or more tests each taking 20 hours or longer to complete set in and I decided to deal with it theoretically. I may run a few tests to validate the theory when i get time though.

image

Using this data I was able to extrapolate how long it would take to fully synchronise down a 25GB mailbox (default Office 365 quota limit).  Given the assumption that there is always sufficient network bandwidth available, the data looks as follows…

image

This data suggests that even if a cached mode client were running over a  500ms round-trip-time network latency connection that it would be possible to fully resynchronise their 25GB OST file within 43 Hours.  This would require a throughput of 1.34Mbits/sec.

One of the customers I am working with is considering a 5GB quota within their Office 365 tenant so I promised them I would consider their case specifically, so here is the data for a 5GB OST sync.

image

Taking the same point of 500ms, this prediction suggests that their 5GB OST file would be fully synchronised after 8.6 Hours, assuming that the client had 1.33Mbits/sec of network bandwidth available.

To perform your own predictions using this data use the following form…

T = ( M ) x (( 0.0114 x L ) + 0.3208 )

Where

  • T = Time to Sync ( seconds)
  • M =Mailbox Size (Megabytes)
  • L = Network Link Latency Round-trip-time (milliseconds)

 

Further tests

This testing sparked a few thoughts in my mind, that I just didn't have time to follow up (and I was being harassed by various colleagues and customers to release the data i had!) … anyway, the following are areas i plan to research further when time allows…

  • What effect does disabling the Windows 7 TCP Adaptive Windowing feature have on Outlook client performance over latent network links?
  • How does Outlook 2007 compare to Outlook 2010 in this test?
  • Where do Cached Mode and OA clients break when just sending and receiving E-mail

 

Final Conclusion…

For me this test data suggests that network latency is not as critical for client performance on Windows 7 with Outlook 2010 cached mode as it used to be on legacy operating systems.  It also shows that the adaptive TCP windowing in Windows 7 goes some way to alleviate the negative effects of network latency.   Certainly for my current projects it has allowed me to consolidate more mailboxes into a central location than i would have initially thought possible and the client involved tells me that end user performance is good.

If you are considering running Outlook clients over high latency connections I would urge you to perform a full user pilot to establish your users tolerance of client performance.  Also some Outlook plugins can have a dramatic effect on how the client performs under very latent conditions.

Neil.

Updated: 17/06/2011