Thoughts from OSCON: Development Practices

by MichaelF on August 14, 2006 11:30am

I recently attended OSCON 2006, and had a lot of fun being there. Microsoft’s presence in the Open Source World is still a novelty (to say the least), so I always got a reaction out of people at the conference when they saw my badge!

We had a large presence there, because we do believe that open source as a development model is here to stay. Bill Hilf was at the conference and Port 25 has some of the interactions he had with open source luminaries Tim O’Reilly and Matt Asay. While Bill was having these interesting conversations, we at the OSSL (Open Source Software Lab) were busy attending the talks at the sessions and collecting “swag” on the exhibition floor! I do have swag from HP, Google, Intel, Dell, AMD, Oracle, ActiveState, Solid and MindTouch but interestingly IBM was missing. Anyone out there have an IBM t-shirt to exchange for our Port 25 t-shirt? (see accompanying pictures)

The buzz in the air, appeared to me to be about open source both as and in business. The talks I gravitated towards (naturally) were about open source development practices. These ranged from taking closed source products and turning them loose as open source projects to driving pure open source development to using experts in a particular domain as contributors for a project not thought suitable for open source.

There was a common thread running through all these talks – the critical nature of development practices. No, there wasn’t anything earth shattering – these were development practices that are accepted as “goodness”. But the forces surrounding open source development made the use of these practices almost a necessity for projects to get of the ground. This is not to say that closed source companies do not follow these practices, but due to co-location, centralized management and other circumstances that go along with commercial development, some of the practices may not be rigidly enforced and the lack of these practices may not impact the product as much. Open source development does not have that luxury (I refer only to successful open source development projects, not the long tail of open source projects that are fossilized on SourceForge and other repositories).

The practices fall into the following categories

Easy “build”

Consider this, you browse to an open source project which is new to you and download it (could be from repositories such as Sourceforge or Codeplex). It doesn’t install and takes a lot of wrassling to run. More often than not, this first impression decides your level of participation. If you can’t find something cool, try it and run it – there are other fish in the open source sea!

The initial install is not the only thing that has to run right, as an open source developer working on a module, adding/modifying some source code, building the source code and running it are part of the iterative process that lets developers be productive. A system which doesn’t make the dependencies transparent and which doesn’t have a build system to include all the necessary files (and NOT include the unnecessary ones) will probably not get good developer input.

The easy build thing has been known for a while. At Microsoft, product groups have the concept of daily builds – if you as the developer “break” the build, you don’t go home till you fix it. In order for this to work, each developer should be able to build the system on his desk easily.

Quick iterative program development in the large without hassle is the name of the game. The very nature of open source development which needs to attract developers to gain momentum leads to a focus on easy builds.

There’s doc (or a blog or a newsgroup) for everything

Most open source developers don’t work in the same building. I am talking about open source developers in the community, not those employed full time by commercial open source companies. (Though most commercial open source developers have to interact with community developers on their virtual team). This means you can’t walk to the next office and ask the developer about how the API call really works!

It follows that at any time of day or night, answers to questions like “how does the API call really work?” should be available through internet accessible means. This could be doc, a newsgroup, a wiki , a blog or any other easily accessible repository.

Documentation by developers, you ask, isn’t that a mythical being?

That’s exactly the point – open source developers do write docs, they just don’t recognize that they are doing so. In order for an open source project to truly take off, education of new developers is a must - both when they are viewing code and when they are looking at documents explaining interfaces, how things work and the meaning of life! Ok, maybe the last thing is not strictly necessary, but it does make reading documentation much more fun. Who wouldn’t want to work on Ruby after reading “why’s (poignant) guide to ruby”?!!!

Every lifecycle stage and artifact is important

The way you work your way up the “committer” chain in open source projects is to prove yourself useful. The path to building credibility is to write documents, find bugs, review codes and make yourself useful in a pretty stiff meritocracy. Even when a developer achieves the golden “commit” privilege they continue to participate in those activities.

Not having departments with people exclusively devoted to test, doc or reviews makes the development of a “caste system” difficult! Development managers cannot put pressure on test managers to shortcut tests – because the development managers and test managers could be the same person!

This is a little bit more subtle than the “more eyes make all bugs shallow” argument – that is only true if those eyes don’t think of looking at bugs as work that is to be done by other eyes! 

This is even truer of documents and education – it isn’t some tech writer with expertise only in writing who writes the important documentation, but the luminaries of the community. When the “Gurus” (which in Sanskrit means teacher) do what they are meant to do – then nirvana is attained (I loosely paraphrase from the Bhagavad Gita!).

Sprints not marathons

Consider having developers in the US, UK, India and Australia – when is the best time for a meeting? When it’s morning in the US, its night in India – and who knows what time it is in Australia? Software companies whose code is all developed by their own employees can have coordination meetings on a schedule decided by someone – not so in open source. This means that coordination can’t be complex and long drawn out.

So open source makes use of XP principles, work on a small feature that doesn’t take more than a month (ok, so it isn’t the extreme XP where the feature shouldn’t take more than two weeks!). Based on community pressures, priorities can be decided. Longer term projects are either done by a single person or by a co-located team (by commercial open source companies for example).

That means planning horizons are small, and mistakes can be rectified without huge loss of time. Releases happen when there is a critical set of features ready. The community is able to get their hands on new features early and give early feedback, which further cuts down the time for stable development.

Of course this means that customers are running hard just to stay in one place, if they accept all the releases! But at least they have the choice…!

Visibility into EVERYTHING

Open source is not just about the source being visible. More people do look at server side code, but even on servers the number of code readers is small compared to the number of contributors.

Visibility in open source is about everything – how many and what bugs are there, time to resolve bugs, prioritization of bugs, who is contributing what, what comments were made about whose code, whose code was included and whose wasn’t etc. etc. etc.

This not only acts as a great feedback mechanism to users, it provides for real and open debate about priorities and execution. As long as the project is handled on a rational basis, people can predict the state of the project. They can anticipate when a feature they want or a bug they have will be fixed. It also allows users to submit code to fix bugs for their own problems and see it transparently go through the system.

My full time job at a previous position was mediating features for a group of customers (numbering in the 10s). This required the full energy of a team, which was not a development team, to gather this information in a closed source environment and then disseminate this information. Of course there were mistakes in information gathering and communication, since the customers only got a view of the project through an intermediary. Building trust with the customers took the better part of the year and resulted in a development process that was not as efficient as it could have been.

So flame wars notwithstanding, visibility into everything is an advantage for open source projects.

Community, Community, Community

In order for developers to be productive they have to communicate with whom they want, when they want and get back what they want. This means there is a burden on the open source product’s leaders to make sure that this responsiveness is part of their community. The artifacts used are answering 250 e-mails a day, having IM on all the time and putting systems in place that make this possible. One open source company I know of uses categorization software just so that the appropriate person can look at an e-mail and has fast mail systems that allow sub-second previews of the e-mails!

What this gains the company is encapsulated in a quote from an OSCON presenter – “When we had a closed source product people worked 9 to 5, but with open source there is so much interaction with the community that our developers are strongly motivated to work on finding solutions and building features and they are much more productive!”

Actually come to think of it, there isn’t a thing here that I have said that wouldn’t work where I work! In fact these processes are already at work at Microsoft. I am not only talking only about the work that we are doing at Codeplex, releasing source for products such as Power Toys for Visual Studio Collection available on Codeplex or the WiX tool available at Sourceforge but also about the code sharing internally within Microsoft. Since Microsoft does both platform and application development, application developers often need and have access to the bug databases and source code of platform level components. There is a lot a give and take between teams of users within Microsoft. This visibility has also been expanded to users such as academic, government and enterprise users under license agreements with Microsoft.