A Look Inside Microsoft’s Open Source Software Lab (Part 2)

by admin on April 21, 2006 05:35pm

A Look Inside Microsoft’s Open Source Software Lab (Part 2)

In “A Look Inside Microsoft’s Open Source Software Lab – Part 1,” we examined Microsoft’s motivations for building the lab, its development, and some of the work done there to ensure interoperability between Microsoft software and UNIX/open source software. In part 2, we discuss some of the lab’s other areas of focus and how that research is helping Microsoft to determine current trends and predict future direction for open source software.

When Bill Hilf was hired to build Microsoft’s Open Source Software Lab in 2004, he knew from the start that working and testing interoperability between Microsoft software and open source solutions would be a large part of the job. However, that’s only part of the lab’s mission. Another area of focus is helping to make IT professionals with UNIX/Linux skills more proficient in a Windows environment—and vice-versa.

“As a UNIX and open source guy, one of the biggest hurdles for me in walking into a Microsoft shop was the lack of crossover between my skills and those needed to manage the Windows platform,” says Hilf. “While there are a variety of aids that can be employed, many of those are migration tools for moving an application from one platform to another. What I needed was an integrated technology that could help me speak a familiar language on a Windows server.”

Helping to examine and test such a solution is precisely what the lab is doing through its work with new technologies that are being developed in the Windows Longhorn Server timeframe. ‘Monad' is one such new technology, a powerful next-generation command line environment that runs on the Microsoft .NET Framework.

“The vision for Monad is to help automate local and remote administration by delivering a consistent, fast, and composable command line scripting environment that spans the Windows platform,” says Hilf. “Windows and Unix and Linux are all different operating systems - so there is no magical ‘layer’ where they all just work the same.  But the ability to compose an environment so that it looks and behaves similar to what I’ve learned in a UNIX/open source world gives me better control over a Windows environment—and makes it easier to move between UNIX and Windows.”

One of the things the lab is doing with Monad involves writing plug-ins using a popular open source tool called VI Improved (VIM) to call on Monad. One researcher at the OSS lab used VIM, a text editor, to write a plug in that calls on Monad to digitally sign a script when the script is ready to be deployed.

“Such an approach allows VIM users—a large population in the open source/UNIX world—to continue using their favorite tools,” says Hilf. “We use Monad for  performing administrative tasks, and are looking at a number of similar scenarios that can help the UNIX/Linux user community feel more at home with the next-generation Windows command line and scripting environment.”

Using Science to Test Perceptions

Along with software and skill-set interoperability, another mission of the lab is to test common software beliefs, relying on scientific analysis to determine the truth. “We hear a broad variety of assumptions around open source and Microsoft software,” says Hilf. “In the lab, we test those perceptions scientifically to determine what’s real and what’s based on ideology or perceptions.”

The lab also tracks other dynamics of open source software, such as how different the various commercial distributions of Linux are from their original open source projects. Figure 1 shows the number of files changed between the original open source Linux kernel and the same version of that kernel as shipped by a commercial Linux distribution.

“We’re seeing changed files increasing with each consecutive release of a commercial Linux distribution,” says Hilf. “Tracking such data allows us to better understand the community versus commercial dynamics at work in the open source model.  It also helps us make better decisions, such as our recent announcement on supporting Linux in Virtual Server 2005 R2 – this type of data helps us understand what the support differences may be between Linux distributions.”

The lab also is attempting to quantify the breadth of the open source ecosystem so that Microsoft can better understand market trends. For example, in August 2005, researchers began counting the number of supported PC and servers for the Red Hat Linux and Novell SuSE Linux operating system products and compared those numbers against Microsoft’s own supported hardware numbers for Windows.

“This type of data helps us understand the size and growth of current market-leading Linux distributions—and how Windows compares,” says Hilf. “It also gives us an indication of what systems customers might be deploying, which in turn helps us to determine the hardware configurations required in our lab.”

Diving Into the Sociological Aspects of Open Source

Another area that the lab studies closely is the phenomenon of community-based software development. About 20 percent of lab resources are devoted to studying the community development model with the goal of learning in order to identify opportunities for working with the development community at large and improving on existing processes.

“As engineers and technologists, we’re fascinated by the community development model,” says Hilf. “We want to understand how testing is done; which tools are used; how test cases are written; how bugs are filed, tracked and regressed; and what type of training testers are getting. We approach the community development model objectively so that we can fully understand the pros and cons of the model from a scientific perspective, without the surrounding philosophy and hype.”

Such analysis has helped Microsoft find new ways to think about its own development projects—such as Microsoft Shared Source (https://www.microsoft.com/sharedsource) and how the company can take better advantage of the community development process. Educators from the lab work with product teams to ensure they’ve thought through all aspects of the process and the types of issues that are likely to surface.

“By engaging with the open source community, Microsoft has been able to mature its own thinking around how to work with developers who are building software using different development models,” says Hilf.

Because of such involvement, the lab has become a gateway for open source developers to engage with Microsoft. “We’ve seen a steady increase in the number of people who are communicating with us from the open source community, expressing interest in having real technical discussions,” says Hilf. “I see this as an extremely positive development in that Microsoft and the open source community have traditionally been regarded as separate realities. The very idea that there might be some bridge-building going on at this level is very encouraging.”

Looking Forward: The Future of Open Source

To better understand the evolution of open source software and where it’s headed next, the lab is also involved in the trending of open source software projects. Through such study, Hilf hopes to answer questions such as whether certain open source software code bases are getting simpler or more complex, whether the number of defects is increasing or decreasing, and whether the code is becoming more or less efficient.

“If you look at only the code, open source software appears to be growing fairly linearly,’ says Hilf. “As the amount of code increases, complexity grows and, of course, with added complexity comes more defects. This isn’t surprising, nor is it a knock against open source software—it’s how all software evolves over time.  The real question isn’t about who has more or less bugs today, but what is the model to not just write code, but to sustain quality software.”

Another trend the lab is investigating is the increasing commercialization of open source software, which directly impacts how open source software is conceived, developed, tested and distributed.

“Over the past five years, more and more contributions are coming from developers who are employed by a commercial entity that either directly makes money from an open source project—such as MySQL or JBoss—or indirectly through hardware, commercial software and services, such as IBM, Novell, and Hewlett-Packard,” says Hilf “To those in the open source developer community, this isn’t news. But to those in the broader market, the extent to which open source software development has become commercialized is often a big surprise.”

A third trend being looked at by the lab is the market for open source software—that is, who the “end customers” of such efforts really are. “I think it’s fair to say that, at least so far, open source software has largely been developer-driven—that is, a good deal of the code is system software that’s being used by other developers or system administrators. Therein lies a key difference between many commercial and open source development models: commercial software companies design and build software to serve a customer need, whereas open source software is largely designed by and for developers and technical users.  You can see this easily by looking at where open source has been most successful.  It’s a fascinating area that we’ll continue to investigate, and it helps us to understand where we might be successful in our community development efforts.”

Adding Value Through a Balanced View of Trends

Through its work, the Open Source Software Lab at Microsoft continues to provide a real-world view on the  maturity and evolution of the open source software movement. By exploring the dynamics of open source software in a way that relies on scientific methods and hard technical data, the lab is benefiting both internal groups at Microsoft and customers who have asked it to look into common open source software questions and issues.

“In some areas, the phenomenon of open source software has become quite powerful and has enabled rapid growth,” says Hilf. “It shows that a variety of software development models can and will coexist into the future. In fact, we’ve found that popular open source software server applications such as JBoss and SugarCRM have a large and growing business on Windows. By continuing to walk the line between cooperation and competition, we’re confident that our efforts at the lab will benefit Microsoft, its customers and partners, and the open source community. It’s an exciting time for everyone.”

Related Links:

Monad Beta:  https://www.microsoft.com/technet/scriptcenter/topics/msh/download.mspx
Shared Source:  https://www.microsoft.com/sharedsource