Customer Experience Improvement Program in EBS

This is a continuation of a previous post where I spoke about the role of Windows Error Reporting (WER) in Essential Business Server (EBS).

Today, I'll be discussing Customer Experience Improvement Program (CEIP) in EBS.  Before working at Microsoft,  I never really understood what this meant.  Various Microsoft applications ask for everyone's participation during setup or later in the program.

A cursory examination of the legalese text sounds somewhat ominous.  "Send data to Microsoft? No way!" probably answer many users.  I hope that by the end of this post, you will understand exactly what kind of data we are collecting, and why we are collecting it.

Why do we want to collect data?

First of all, why do we have this program, and why do we want this data?

At Microsoft, we try our best to determine which features and UI design would best suit user needs.  We perform usability studies, conduct other research, and take in customer feedback.  However, the best feedback we can get on our products is from the actual users.

Have you ever hit an error dialog box that could have been avoided if only the UI were designed differently?  Have you ever wished you could tell the developers which features you use the most, and which ones you avoid because they are troublesome to use? That is why the CEIP program was developed at Microsoft and why we use it in EBS.

The Customer Experience Improvement Program (a mouth-full) is exactly what the name implies;  It's a program to improve our products (or "customer experience" as they call it) by collecting anonymous data about how our products are used by actual people.  We try to collect data (the kind of data is described later on) that helps us answer specific questions (called "Business Questions") that we can use to improve the next version of the product.

Here are some of the Business Questions we might try to answer about our EBS Admin Console:

  • What kinds of video resolutions do customers run at?
    • How many customers really run at the minimum resolution that we design the admin console for (800x600)?
    • How many customers use wide-screen resolutions?
    • What's the predominant video resolution used?
    • To what size do customers resize the Admin Console Window?  How many maximize the window?
  • How are customers using our product?
    • Do they use the Admin Console to add new users?
    • Which pages do they use most?  Which Wizards?
    • Which tasks do they run on each page?
  • What kinds of users are using our product?
    • Does more than one Administrator use the admin console at the same time?
    • Is EBS being used with many servers?  Or just our 3 servers?
    • Is EBS being used with how many users?

Knowing how our product is used helps us update the next release to meet the needs of real customers.

What kinds of data do we collect?

We collect many types of data, but it basically boils down to a few basic kinds:

  • Counters:  We count how many times something occurs.  How many servers are in the environment, etc.
  • Flags:  Has a specific error occurred?  Did the user cancel a wizard before completing?  DHCP in EBS enabled, or is the user using a different server/router?
  • Discrete Data:  Which of the 10 options did the customer pick?
  • Strings: Not used very often as there isn't much data that isn't private.  See below.

Every piece of data that we collect needs to go through a stringent process here at Microsoft to check that the data we are collecting is anonymous and non-confidential.  For example, getting string data approved is very difficult. You have to show that it will never contain personal information, a high bar – so we usually choose other ways to help us make product decisions to improve the product.

For example, say we wanted to know if we made the user name text box big enough in our New User Wizard.  Instead of collecting the usernames, which is personal information, we could collect the lengths of user names entered.

Here are some examples of data we might typically collect from a component in EBS, such as the New User Wizard.  (These might not be the exact measurements we collect, but they are measures typical of the ones we make in EBS.)

  • Time Metrics
    • How long did the user take to run the New User Wizard from start to finish?
  • Failure Metrics
    • What errors did the user encounter when using the New User Wizard?
  • Usage Metrics
    • How often does a user run the New User Wizard?

What do we do with the data?

This was somewhat addressed above, but there are a lot of things we can do with this data.

  • Frequency of feature use
    • If a feature is being used heavily, we devote more resources to improving and/or expanding the feature. 
    • If a feature is rarely used, then we either improve the feature so that it is used more, or consider cut future work on it if it looks like no one finds it useful.
    • I've been told that the Office team used their feature frequency data to help decide which/where items should be placed on the Office 2007 ribbons.
  • "Pain Points"
    • Where are users getting errors dialogs and/or prompts?  Knowing where the "pain points" are helps us retool the UI in future releases so that user errors are more easily avoided.

How can you participate in CEIP?

I hope that by now you realize how useful it is to participate in CEIP.  This is one of the easiest ways to give us meaningful feedback about our product (see our privacy policy for which products in EBS are configured) while requiring just about no effort on your part - just opt-in!

In EBS, there are 2 places where you can configure your participation:

First, there is the setup page:

Screenshot of Setup page in EBS

This is the best place to opt-in because then we can gather information about what your setup experience was like, any errors and, possibly, retries you experienced when setting up our product.

After setup, you can configure this setting in the Admin Console:

Screenshot of Help menu being clicked  Screenshot of CEIP dialog 

If you didn't opt-in in setup, you can still start at any time.  We really appreciate the feedback from those that choose to participate!  By participating, you are helping us improve the software you use every day - improving it in the way that helps you the most.

If you have any questions or comments about the Customer Experience Improvement Program or Windows Error Reporting, please leave comments below, and I'll try to respond in a timely manner.