Protocol Test Suites (Part Two)

This is the second part of the two-part series on Open Specifications Protocol Test Suites. In the previous post, we gave an overview of Test Suites and their different components. Now let's go over the prerequisites of a Test Suite environment, the typical contents of a Test Suite package, cover the end-to-end process of configuring and running Test Suites, and analyze the results.

Setting up a Test Suite Environment

The following table lists the tools and software necessary to set up your environment for Exchange and SharePoint Test Suites.

Note: For information on setting up a test environment for a specific Test Suite, refer to the Test Suite Reference Guide for that Test Suite. For more information on the Reference Guide and other contents of a Test Suite folder, see the subsequent section.

* SUT may be replaced in a non-Microsoft implementation

Using a Test Suite

When you download a Test Suite, the package is downloaded as a compressed folder in .zip format. When you unzip the folder, you will see several different folders and subfolders. It is helpful to understand what each one of these folders contains.

Folder structure and contents

A Test Suite folder mainly consists of configuration files (both client and server-side), test cases, and associated supporting files. These can include scripts, methods, properties, source code, technical specifications for protocols associated with the Test Suite, and end-user help documentation such as a Reference Guide, a Test Suite Deployment Guide, and a Test Suite Specification. The folder also contains a Requirements Specification spreadsheet (a list of normative requirements extracted from the technical document) for each protocol associated with the Test Suite.

The following diagram shows how the contents of a Test Suite folder are organized:

Figure 1: Test Suite Folder Structure

Note: For more information on each component in a specific Test Suite folder, refer to the Test Suite Deployment Guide for that Test Suite.

Running Test Suites

The following illustration shows the basic steps of running a Test Suite.

Note: Information for running a specific Test Suite is available in the Test Suite Deployment Guide for that Test Suite, found in the Test Suite folder.

Figure 2: Running Test Suites

Note: The Test Suite Deployment Guide contains detailed guidance on how to configure the client and the SUT, run test cases, and view debug reports.

Analyzing the Test Results

After a Test Suite has run successfully, log files and reports are generated. These files enable you to identify errors and determine what exactly failed in the validation. The following types of files are available for analysis:

• TRX log: A log file produced by Visual Studio. This file contains information about how many test cases were run, and pass/fail results for each test case.
• HTML report: Produced alongside the TRX log, this report contains the verification status (pass/fail) of the protocol requirements.
• XML log: A log created by the Protocol Test Framework (PTF).
  Note: The Protocol Test Framework (PTF) is one of the client components of the test environment described in the first section of this post. The PTF is designed to provide support for the Microsoft Protocol Test Suites for Office and Windows Protocol Interoperability testing. PTF enables you to develop custom Test Suites for your implementation, and provides an interface called ITestSite to represent the environment for Test Case execution.

Figure 3: A sample TRX log file

Where to Get the Test Suites

You can download the Exchange, Web Application Open Platform Interface (WOPI), and SharePoint Test Suites from the Test Suites page on Microsoft Connect. This site is also where you can download the Protocol Test Framework (PTF) and Message Analyzer.

Currently, the following Test Suites are available for download.

Note: More Test Suites are always being added, so this list may become out of date. Visit the Microsoft Connect Test Suites page for the most current selection.

Resources and More Information

The following resources provide more information on Test Suites and Open Specifications.

• Test Suites on Microsoft Connect
• Test tools for Interoperability testing
• Open Specifications Plugfest videos on Channel9
• Open Specifications documentation on MSDN
• Open Specifications Dev Center