Best Practices for: Automated Testing; Career Development; Software Feature Sets (keeping it simple), from Michael Vax: Industry Leading Vice-President of Product Development at Blackboard

This is the next interview in the continuing series of Computing Canada's (CC ) Blogged Down (BD) feature which is highlighted here in the Canadian IT Managers (CIM) forum.

This week we talk with industry leading development authority, Michael Vax, Vice-President of Product Development at Blackboard. In the first blog, we began the interview and profiled Michael's extensive background in development and leadership.

Today I put this final question to Michael:

Stephen: Can you provide commentary on three topics of your choosing?

Michael)

TOPIC 1: We have talked already about the growing complexity of s/w systems and how challenging it is to ensure the quality of software products. Automated testing is a great way to increase quality and make it consistent. I want to share with your readers our experience with it.

It takes more than 33 man/weeks for example to perform the full regression test of the WebCT Learning Management System, which is a highly configurable enterprise online application.

Four years ago, to reduce these manual testing efforts, we started to look at test automation tools. It is relatively easy to start with tools like Silk Test or WinRunner. You record a test script by just clicking your way through an application. This perceived easiness can lead to a trap. As the number of scripts grows you can find yourself spending more time maintaining them than it took time to develop the automated tests. In many companies automation efforts are led by testers who don't have a programming background. But for automation testing to become successful you need to treat it as a development project and do proper design and architecture.

This was the first challenge we faced after half a year into the automation. We addressed it by hiring experienced automation testers and implementing a data-driven approach where test parameters are stored in the database and guide the test execution. To run tests we set up dedicated machines.

After a couple of years using SilkTest we had automated up to 20% of test cases. This is considered a good achievement in the industry. At that point we became victims of our own success. It was taking a long time (up to 3 days for three people) to run all the automated tests. They also required substantial maintenance for each release as there were changes in UI and functionality. Another problem was an instability in running tests. Occasionally we experienced crashes of the browser or the test tool which made it impossible to reliably run tests unattended. This instability, vulnerability to UI changes, and slowness in the execution was caused by the fact that tools like SilkTest emulate a user clicking buttons and links in a web browser.

The WebCT performance lab was using a different tool - SilkPerformer - that instead of working inside the browser records get and post requests and sends them to the server. We rewrote some of the scripts in SilkPerformer and saw great improvements in script execution time. For example, our smoke test execution time was reduced from an hour and half to ten minutes. It also became easier to maintain as SilkPerformer tests don't depend on the positioning of UI elements. Everything has its price of course as SilkPerformer will not find a pure UI bug. We still thought that this was a good trade-off in many cases as most of our application logic is implemented on the server.

At the same time we extended our automation team by hiring three junior Java developers and started to implement automation scripts in Java. This opened new opportunities. For example, we implemented a test that emulates hundreds of students taking a quiz at the same time, providing correct and incorrect answers to questions. At the end, the test also checks that grades are calculated correctly. It's all parameterized and can be executed in a matter of minutes. Some of these test cases are impossible to execute manually.

I want to mention another non-technical problem that we needed to overcome - initially testers were not trusting results of automation testing and continued to do manual testing even after the automated test had been run. To overcome this we started to get manual testers involved in writing requirements for automation work and trained them to execute scripts themselves. The problem was solved completely when we created custom scripts to report results of automation test runs into the same test case management system that is used during manual test execution. Now a tester is able to run a report on test coverage that combines results of both manually- and automatically-executed tests.

To summarize, you can start easily with automation testing by using one of the automation tools. If you want to do more than just ad-hoc scripting start treating automated testing as a development project, define coding guidelines, build a framework, and use a data-driven approach for script parameterization. If the UI of your application does change often, move away from UI-based test automation and start to write test applications in real programming languages that exercise product functionality through internal or external APIs. In all cases, make sure that results of your automated tests are converged with manual testing results.

TOPIC 2: As you know, I'm interested in career development. As a manager I mentored many technical people in choosing the next step in their career. The most difficult move is to switch from a technical position to a management one and I know this from the first hand experience. It is quite scary to stop being a hands-on person.

It is not enough to be good technically to become a good manager. However, having a technical background is a great help in making right management decisions in IT. People who are thinking about advancing their career into management should do it for the right reason. Go for it if you enjoy working with other people, want to be involved in making strategic decisions, do things on a bigger scale, and have more room to implement your ideas.

I advise people who want to move in this direction to take business and management courses, learn project management, read books on s/w development processes, and learn to see the big picture. If you are not sure that a management career path is for you, you can try a taste of it by accepting a team lead's position, or volunteer to lead a small project.

My interest in the career development led me to found CareerShare.com. Its goal is to create a community where people can share their real life career stories and get support, advice, and motivation from others. The real life is so different from what is written in a college recruitment brochure. People are making fascinating career turns and switches. I believe that collecting and sharing this information will be a great help to those who are just starting their journey as well as to somebody who looks for the next step in professional life.

TOPIC 3: Pragmatism in making trade-off decisions or how to make s/w good enough

Quite often we see overcomplicated products overloaded with rarely used features. I'm a firm believer that business success comes to a company that is good at making pragmatic decisions about what functionality to add to its products and assessing its benefits versus cost of development and maintenance.

We need to realize that most of the cost associated with a particular functionality occurs after it was initially developed. If a developer writes code that is barely used we are still going to pay for it throughout the life of the system which can easily be 10-20 years. As Mary Poppendieck says in her Lean Development book - "Complexity is like a cholesterol, it clogs up the arteries of a code base, as well as an organization; ... If we want our code to have lasting value, we will put top priority on keeping our code base simple, guarding vigilantly against the creeping crud that plagues so many software systems. "

The same applies to technical decisions. Quite often designers advocate using the latest and greatest technology just because it excites and challenges them. This may lead to overcomplicated solutions that are not really required to achieve business goals of the application. Let's keep it simple!

Stephen: Closing comment: Michael, we very much appreciate your dedication in providing this interview. Thank you for sharing your experienced and deep insights with our audience.

Michael) It was my pleasure Stephen.
________________________

I encourage you to share your thoughts here on these interviews or send me an e-mail at sibaraki@cips.ca.
________________________

Computing Canada (CC) is the oldest, largest, most influential bi-weekly business / technology print publication with an audience that includes 42,000 IT decision makers in medium-to-large enterprises. For more than 30 years, Computing Canada continues to serve the needs of Canada's information technology management community - you can request your free subscription at: https://www.cornerstonewebmedia.com/plesman/main/Subscription.asp?magazine=CCA.

For the latest online business technology news go to: www.itbusiness.ca

________________________

Thank you,
Stephen Ibaraki, FCIPS, I.S.P.