How does usage data improve the Office User Experience?

Hello, my name is Tim Briggs – I am a user experience (UX) Researcher in the Office Design Group (ODG). Usage data is one of the important feedback mechanisms we look at to understand how Office is being used in the real world. Answering questions like “how frequently is a command used?” and “how many files contain this feature?” are critical parts of our entire engineering process.

Today, I’d like to give you a better sense of how we collect, interpret, and design based on usage data. To get the most out of this discussion, some background might be helpful to you. I encourage you to check out Shawn’s post on Designing with Customers in Mind post—it will give you an idea of our research and design process. I’ll also point you to Peter’s post on Data Driven Engineering—it introduces the Customer Experience Improvement Program (CEIP), which is the source of the usage data I’ll discuss.

What types of usage data do we collect?

We collect two main types of usage data: command usage and feature usage. Each gives us a different perspective but the goal of analyzing both is to find patterns of usage in customer data. It’s also important to note that because the data are coming from CEIP, they are anonymous and contain nothing that can identify individuals or their content. But seen in aggregate, general patterns of usage emerge that help us explore, confirm, or challenge our understanding of how all of the Office applications and services are being used.

How do we use Command Usage data?

Command usage data (aka “click data”) tell us how a specific command is used. For instance, take the example of the Paste command. If we combine all of the clicks on Paste, we can mine that data to get a better understanding of:

  • How frequently is it used—should we make it more or less prominent?
  • How many people use it—what’s the impact of improving the experience?
  • What’s the most frequent way of accessing this command—can we make it easier?
  • Does this command occur as part of a clear workflow—how do we better support that?

clip_image002

The Paste command in Word 2010 – command usage data helps us understand how it’s used

Command usage data also tells us about how different parts of the user interface are being used.

  • Within a ribbon tab, what are the most used commands—should we rearrange them?
  • Across tabs, what commands cause the most tab switches—where should the command live?
  • How and when do people use right-click contextual menus—how can we improve that mechanism?

clip_image004

The Home tab in Word 2010 – the arrangement and size of buttons is informed by command usage data

This is basic but super useful information…and a bit daunting. Did you know that in Word alone there are over 2,000 commands? Want to know the most common Word command operating on text? Paste.

How do we use Feature Usage data?

Feature usage datais a little more difficult to describe. Let me try. Think of a “feature” as a general capability of Office, like Tables in Word. There might be many commands associated with using tables (e.g., Insert Table, Delete Row, Move Column…). While we might be interested in each of those commands, often we want to understand use of the feature at a higher level. This requires feature usage data—counters that we build into the feature to answer specific, predicted questions:

  • How many files contain a Table—what is the impact of improving the Table feature?
  • How big is the average Table—should we focus on improving large or small Tables?
  • What are the most frequently used Table styles—what design choices should we provide?
  • What other features are used in files containing Tables—how can we improve the interaction of different document parts?

If you want an interesting exercise, think of an idea you have for improving Office. First, use Send-A-Smile to tell us about it. :) Then, think about what information you would need to know about how that feature is used in order to create and validate your design for all of the users that will be impacted. That’s your feature usage data. And here’s a tip we use with all of our teams: sketch out the graph of the data which you expect to see once implemented—this helps you think about the form of the data to collect and sets your expectations for the final data to help you spot errors in your initial assumptions.

This is just a sampling to give you an idea of the breadth and depth of typical usage data we use across our engineering stages, from initial ‘what-ifs’ to late stage ‘Beta use is showing we have to change this.’

The Story of Paste and Usage Data

Let’s look at an example. Usage data tells us that Paste is the most frequent editing operation (for instance, it’s used almost two times more frequently than Bold). We also know from our other research methods (field visits, usability labs, focus groups, Send-A-Smile, etc.) that sometimes reformatting pasted content can be painful. We feel it ourselves—it can be hard to get the new content to look right.

As we began working on Office 2010, we wanted to improve the Paste experience. In order to do that we needed to understand more about how Paste is used. We looked to the usage data to provide some context. Below is an example of some of the ways usage data informed the engineering of what would eventually become the Paste Options gallery, which Mirko explains in his post on Live Preview Paste.

Question

Data used

Results

How many users would this impact?

Command Usage:

  • % of users using it
  • % of sessions in which it’s used
  • # of times users used it in total
  • # of times users used it per session

Paste is used by almost every user in almost every session of Word. In fact, it occurs more frequently per session than other related commands…like Cut or Copy.

How are Paste and Paste Recovery being used?

Feature Usage:

  • Paste. Knowing we would always want to understand more about how people paste, we had instrumented the Paste feature to count the number of times it was used in different contexts.
  • Paste Recovery. This is the little on-object UI (OOUI) widget that appears after you paste. Analyzing its use helps us understand the final format of many pastes.

clip_image006

clip_image008

The result was the chart above. Note how frequently content is pasted from a different document or different application—those sources probably have different formatting, explaining the need for the additional formatting options provided by Paste Recovery.

How do users get to the commands?

Command Usage:

  • # of times it is accessed from the ribbon, from its shortcut (Ctrl-V), from the right-mouse menu, from the Quick Access Toolbar, etc.

Predictably in Word, the most frequent is Ctrl-V by a wide margin, with the right-mouse menu next.

We used this data to explore 1) improving the ease of access, the contents, and the interaction model of the Paste Recovery widget for keyboard users; 2) putting the Paste options directly in the right-mouse menu, for those users who are accustomed to pasting via the context menu; and 3) we also thought about how we can expose the improved Paste experience in the Ribbon. After iterating on several designs in the usability lab, you can see the result in the Office 2010 Ribbon, right-mouse context menu, and the Paste Recovery OOUI widget:

clip_image009

What do people do after they paste?

Command Usage:

  • List of the most frequent commands which follow Paste.

Other than continuing to type, the most frequent commands relate to formatting (changing font size, color, appearance). The next most frequent are another Paste or an Undo.

We used this data to understand what options needed to be available in the Paste Recovery options. It also clearly signaled that people are not satisfied with Paste results since they were often making “manual” formatting changes (or undoing the action) immediately after pasting.

Which Paste Recovery options are most used? Do we have the right defaults?

Feature Usage:

  • % of times each Paste Recovery option is chosen.
  • % of times different Paste Recovery options are evaluated but not chosen.

clip_image011

The result was the chart above. It shows how frequently each Paste Recovery option is used. KeepSourceFormatting is the default and comparing that to the number of times Paste is used versus number of times Paste Recovery is used, we have the right default. Note that the other options in the Paste Recovery widget are arranged in order by frequency:

clip_image013

Armed with the usage data above, we explored different designs, iterated on them, and arrived at the current design that Mirko explains in his post. And now, as we receive usage data from the Office 2010 Beta releases, we continue to analyze the usage of this and the other significant improvements we made this version.

So, I hope this discussion has given you a sense of how we use usage data to improve the Office user experience. It’s been a game-changer for us…thank you for providing us the data.

Tim Briggs, UX Researcher, Office Design Group