LOB Workload Testing: Single Unit, Load, Stress, and Smoke Testing

  • Article

Testing the performance of a particular mission-critical workload on your Line of Business application (SAP ERP or SCM, Dynamics GP, Oracle Siebel CRM, Peoplesoft HR, JDE, and so on) is part of a smart IT organization’s life. Based on your IT organization’s budget, headcount, bandwidth, and other available resources, though, the particular type of workload testing you have the time and money to execute might vary.

 

We all know the drill – sure, it makes sense to load test SAP or any enterprise application prior to “going live” or after making significant upgrades in functionality or the application’s technology layers. After all, the risk of not doing so is enormous, and is reflected in poor response times, missed SLAs, unplanned downtime, and even pointed articles in the Wall Street Journal. But complete end-to-end load testing takes time, and isn’t cheap, especially when it comes to testing cross-application business processes or evaluating the impact made by thousands of concurrent SAP users. Indeed, with investments in people, software, and testing hardware exceeding hundreds of thousands of dollars a year, even some of the most risk-averse LOB applications IT shops “ignore” load testing, choosing to cross their fingers or pray for smooth sailing instead.

 

Don’t let the challenges sway you into avoiding load testing altogether, though. There are quite a few alternatives to expensive traditional methods of load testing, four of which are compared and contrasted here. Before moving forward, set aside any preconceived notions you might have, and continue reading with an open mind and a willingness to consider a few alternative methods and approaches to LOB workload testing.

Four Types of Workload Testing

Load testing is a fairly misused term. To many, it’s synonymous with “stress testing” or “performance testing.” Others lump it into user testing or some kind of technology-specific validation testing. All of these have merit, and I’ve used them interchangeably myself over the years. For our purposes here it makes sense to define load testing, though, as one of four different types of what is better described as SAP workload testing, as follows:

 

  • Single-unit testing
  • Average workload load testing
  • Peak workload stress testing
  • Maximum sustainable workload smoke testing

 

These four types of workload testing are easier to differentiate when they’re mapped relative to one another; refer to Figure 1 below.

Figure 1 : The Four Types of Workload Testing

Single Unit Testing

The simplest type of workload testing is sometimes called single unit end-user testing. It's very basic but can be useful. Single unit testing is appropriate for very simple system changes where the maximum acceptable load can be easily extrapolated from the performance of a single end user. For example, a customer changing a bit of functionality in a single module within their SAP ERP system might be a good candidate for such an approach (maybe...). It might be as easy as running a sample transaction before it's changed, running it again after the change, and comparing run-times or the delta in "wall clock" time. This is often a good approach for testing batch job changes as well.

Average Workload Load Testing and Peak Workload Stress Testing

The next two types of workload testing – “average workload” and “peak workload” – may be more appropriate when thousands of users are hosted by a system (the former case), or validating SLAs for mission-critical business processes (the latter case). Such workload testing requires an investment in application-aware virtual load testing tools like those available from HP (used to be Mercury), Compuware, Borland (used to be Segue), and others. My personal favorites are LoadRunner and SilkPerformer. Freeware like OpenSTA is really useful too. Alternatively, IT shops may conduct real-world average load/peak stress testing by coordinating the activities of actual users in an ‘off-prime’ time-frame (prior to virtual testing capabilities, this was a fairly common though expensive and inconvenient option).

Maximum Workload Stress Testing

The fourth type of workload testing – “smoke testing” – is rare for most customer environments. But for systems where raw processing speed is of the essence, and time saved (i.e. reducing the execution window for key transactions or business processes that enable key decision-making) equates to a huge pile of cash, smoke testing may be the key type of workload testing to conduct. Smoke testing is all about identifying potential risks so they can presumably be mitigated. In my eyes, it's the coolest kind of testing, too. It's all about pushing a workload to the max to identify the first bottleneck that actually throttles system capacity. Picture pushing your servers and disks until they're "smoking" from all the work....you get the idea.

Conclusion

Based on an IT organization’s budget, headcount, bandwidth, and other available resources, a particular type of workload testing is called for. I suggest starting with single unit testing and then obtaining a test tool to do some kind of minimal average load testing. An excellent practice is to perform load testing as part of your quarterly promote-to-production process. In this way, you’re more likely to catch the really big showstoppers before your end-users do! That’s certainly a great place to start…. J

Best to you! (and for my friends in the States, enjoy Thanksgiving and Black Friday!)

George