A Tale of Two Articles (on Office 2010)

  • Article

I found this rather interesting and had to share.  I’m not a Wired reader, but I ran across two Wired articles as “related” links to other Office 2010 info gathering I was doing.  The first one turns out to be a reasonably interesting article called How Microsoft Crowdsourced the Making of Office 2010 which I’ve provided the link to.  The other is an article that purports to be a review of Office 2010 that’s a prime example of almost everything I find wrong in online reviews – jaded, parochial, makes generalizations about anecdotal issues, makes comments and conclusions that are not always informed, accurate, or supported by any objective evidence (so I won’t be including that link). 

Allow me a second to editorialize -- Readers of online reviews beware, as I know you are – just because an article or post gets internet “air time” and may be published by a well-known sponsor does not mean that person has the expertise you would hope for in the subject matter reported on.  The only thing worse in all the respects above, in many cases, is the comments.  Which is why I’m writing this post.  I typically don’t read many comments because, by and large, they tend to be even more jaded, parochial, uninformed. and unsupported than the article they are commenting on.  But as I mentioned above, I found the juxtaposition of these articles/comments interesting so I thought I’d share and hope you agree…

Regarding the first (linked) article, Brian X. Chen, the writer, gleaned some interesting data points from discussions with a Microsoft usability engineer (who had been doing usability research for MS for 10 years).  Here’s some of things brought out in that post that I found interesting, and I hope you do as well, AND, as always I share this to help you with some “ammo” for customer conversations that you have as partners and distributors.  I’m guessing almost every one of you has had a least one client/customer who has complained themselves or heard that the “ribbon” UI in Office is not as good as their old menu UI.  So let me hit a couple of highlights of this article:

  • First, Microsoft has a whole research department dedicated to usability testing, and has had that going on for many years (at least well before I came onboard 13 years ago in 1997).
  • 9 million people downloaded the Office 2010 beta and Microsoft collected over 2 million comments – this is what the writer refers to as “crowdsourcing”, an interesting term he presumably coined.
  • Another 600 people participated in Virtual Research Labs where their usage could be observed; which prompted the engineer to state: “we know from watching them work that they really need it [certain features].”
  • Microsoft included a feature in Office 2010 called “Send a smile” to facilitate even more feedback, which is part of why they were able to collect the 2 million comments, and over 80k of those folks included contact info so Microsoft could follow up.
  • To analyze the Send a Smile feedback, Microsoft built a database and programmed algorithms to classify and tag comments under certain categories, while filtering out biased feedback and useless drivel. From that point, researchers manually read every single comment to determine necessary tweaks and additions.
  • A major new feature birthed from customer feedback was the PowerPoint online broadcasting tool for users to share presentations by simply sending around a URL
  • One of our Office CVPs stated: “We were making many decisions not based on what others were doing, but on what customers wanted us to do…We did research on customers that led us to the path.”
  • The virtual lab is effective in helping researchers understand how and when people become confused and unable to finish a task or resolve a problem.
  • Beyond the virtual testing, Microsoft usability engineers also worked 1-on-1 with live subjects to test early versions of the Office Web Apps (OWA), which was crucial so we could get live feedback.
  • Of course, no product comes out perfect as a V1 -- and even most mature products will never be perfect in the opinion of any given individual (that last thought was strictly mine and not a part of the post).
  • Points out that there are 360 million Windows Live users to help crowdsource improvements in OWA.

So that’s a summary of the first article, and should be nothing new to those of you who have read my blog for a while, as I’ve mentioned our research, not to mention our support resources, on many occasions and pointed out that this is another reason why a bet on Microsoft technology is a better bet that on other technologies, e.g. open source, or even other companies that cannot or do not invest in research, or gain product feedback thru support tools such as error reporting instrumentation.

So the real reason I wrote this post was because after reading the above article, I read the so-called review, and since it was rather poorly done, I thought I’d read some of the comments to see if anyone else picked up on some of the limitations of the article.  As you might expect, the predominant strain in the comments was anti-Microsoft, many of them railing about the ribbon and extolling the virtues of menu systems (because that’s what they’re familiar with) with nary a clue about the real efficiency gains that those of us who have adopted and embraced the ribbon are realizing.  And then there was this exchange that caught my eye since I had just read the article about our UI research and usability testing. 

Posted by: idioteque2k -- Why is it that when Microsoft releases something with tons of features, it's regarded as a negative thing... but if someone else does (OpenSource, Apple) it's "feature-rich" I don't understand why someone would bother writing the article, with an ap...

Posted by: btinc -- Because when Microsoft piles on features, they do it in a way that makes the product more difficult to use. The Windows interface has no consistency, and no usability testing. Options are buried, interfaces are ugly....[my note: he/she should read the article above, and for sure doesn’t understand the whole concept of the ribbon UI, and perhaps should consider doing something more constructive with their life.]

And as for the second (not linked) article, rather than going thru the litany of miscues or issues I have with the review, let me just highlight one as an example and you can draw your own conclusion about the validity of the rest of the review from there.  As you know, in my job, I talk to Microsoft partners on a regular basis.  In fact, I mentioned a recent post that I have been on the road doing Office 2010 OEM Reseller Launch events over the last month.  Many partners, as technology advisors, are also early adopters of technologies and products, especially mainstream ones like Office.  I’ve talked to literally hundreds of these folks over the past few months.  One of the things that typically gets the most positive comments is the printing in Office 2010 – I’ve not experienced, nor had any of the hundreds of folks I’ve talked to ever mentioned that they or any of their clients have experienced printing issues with Office 2010, in fact, a few have even listed that as their favorite new feature.  Yet, here’s what our intrepid writer of this review has to say (remember with absolutely no support from any other resource than his own anecdotal experience as far as I could tell).  “If there's one major problem with Office 2010, it's printing…You can't keep from previewing a document before printing [my note: apparently this “expert” reviewer doesn’t know about the Quick Print option so he draws an uninformed and incorrect conclusion there], and rendering a graphics-heavy e-mail can take up to 15 seconds before you can even push the Print button. [my note: I’ve never experienced or had anyone else mention this kind of delay, so he makes a generalization about an ostensibly anecdotal issue on his system] For busy admin types, this is a deal killer. [my note: a very jaded, and unnecessary, comment there]  What's worse is that in Outlook, printing is erratically implemented, so if you're printing an e-mail you already have open in preview mode, sometimes Office will print a list view of the entire email folder by default. [my note: never had anyone mention this either, but hey I’ve only talked to a couple hundred folks, and have not checked our support queues]  On the whole, printing in this version of Office is such a step back, I consider it fundamentally broken. [so that’s his final assessment of what many of my partners find one of the most improved features in Office 2010.]   If you happen to have experienced print issues (btw, there was no mention of which printer this was or what version or age of driver it may have been using so there’s no way that could have been the issue do you think?), then perhaps you’ll have some confidence in his other findings as well, but even so, I’d proceed with a huge shaker of salt nearby.  Btw, he went on to state that “Web Apps won't be challenging Google Docs (or any other online document editor) anytime soon, as it's uncommonly convoluted and buggy.”apparently without the slightest concern, that many business have, about the lack of 100% document rendering fidelity and round-trip editing that only Office Web Apps can provide.  So hopefully I’ve made my point and saved you a read on this one.

OK, now that I’ve warned you, in the interest of fairness, here’s the link to the second article if you must.  Oh, and for the record, re his railings on spam filtering, I would suggest that spam filtering is best done at the server level not on the client itself, but as I like to say, “you make the call”.. 

Cheers,

Ron