Reducing test costs is often the first thing organizations think about when considering adopting test automation. After all, software testing is incredibly expensive; it represents on average 23 to 35% of overall IT spending, for example the latest World Quality Report. Automate testing allows you to focus on more interesting and value-added tasks, while performing routine checks faster, more frequently and with increased precision.
Clearly, the value is there. However, even open source testing tools require an investment in resources, and that requires buy-in from someone higher up. How do you convince the business that test automation is worth the time, effort and cost required?
The typical approach is to do a simple ROI calculation: estimate how many hours of manual testing you can free up, then multiply that by the testers’ hourly rates. It is certainly a good place to start. But the elements of a strong business case for test automation include time to market, quality / risk, and a much broader assessment of ROI.
1. Time to market
Time to market depends on how quickly you can get a new feature or update in front of the end user. The longer it takes you to release new features, the less agile you are and the harder it becomes to meet dynamic customer needs and emerging market trends. Depending on the nature of the application, delays can negatively affect revenue, operational efficiency, and / or your competitive advantage.
Since tests are regularly cited as main source of delivery bottlenecks, it seems fair to say that time to market is highly dependent on the efficiency of the tests (how fast you can run your tests).
Test efficiency, in turn, is based on the time it takes to run an automated test run (including preparation, execution, and analysis) versus a regular test run (again once, including preparation, execution and analysis).
The greater the difference, the greater your potential to save time to market. And, the more frequently you run tests, the greater the payoff.
Other considerations that affect the effectiveness of the tests include:
- How many tests you need to run: It can be more or less than what you currently realize – test case design methodologies and risk assessments can help.
- Availability of test data and test environments: It is not enough to define automated tests; you need to have continuous access to all the items needed to perform them if you want to reach the finish line faster.
- The point at which testing takes place in the software delivery lifecycle: ‘Shift to the left’ strategies, such as adopting automated API testing and using new technologies to start testing user interfaces and APIs before they are implemented, can have a positive effect on the efficiency of the tests and therefore accelerate the time to market.
2. Risk reduction
The reduction in risk depends on the number of defects that you can prevent from slipping into production. The escaped faults bring a series of misfortunes. You have probably seen the curve showing how it is exponentially more expensive to correct post-release defects than to detect them early. But there are also downtime, support costs, and in many industries, penalties to worry about.
Knowing your business’ risk coverage is essential for assessing risk reduction. Determining this involves ranking the risk of your requirements or use cases against each other, mapping the tests to those risk-weighted requirements, and tracking how well your tests cover your top risks.
It is quite possible to have 10 tests created strategically to achieve risk coverage well in excess of 100 tests created without risk in mind. The higher your risk coverage, the lower the defect density.
3. Return on investment
The third item is the one people usually think of first: How much time and money can you save with more effective testing? With the help of automation, you can run more tests at the same time, or you can run the same number of tests in less time.
You can find a lot of quick and dirty formulas online, but determining them exactly depends on several factors. Make sure you take things like:
- Number of test cases
- Number of versions
- The time and effort required to manually create reports
- The number and types of apps you are testing
- The duration of manual tests
- The time required to acquire / create and prepare the test data
- The time required to acquire / create and prepare test environments
- Business risk coverage of the test suite
- The level of redundancy of the test suite
- Who is involved in the tests (including business experts) and their daily rates
- The total cost of labor
- The cost of repairing a defect that creeps into production
- The stages of your tests (smoke vs daily vs complete regression)
To put the pieces together
To give you a concrete idea of what a business case might look like that took all of these various elements into account, here is a real, anonymized example of a major European food and beverage manufacturer.
Figure 1: This organization has collected a lot of details about their testing efforts. Source: Jori Ramakers
All data points play a role, but let’s focus on the things with the most impact:
- The company’s current risk coverage is 45%. Most organizations that are just starting a test transformation have around 45% to 55% risk coverage, so that’s pretty standard. Many organizations do not calculate risk coverage. This organization has not yet optimized its test suite or adopted a methodical approach to test case design (pairwise, orthogonal, linear development, etc.).
- There’s only one app being tested, but it’s a big one: SAP ERP Central Component (ECC). Automating tests for complex business applications that require specialized domain expertise tends to be more cost effective than, for example, automating tests for simple web or mobile applications.
- The defects found in the test amounted to 650.
- In terms of release frequency, the company has one major release and 11 minor releases per year, plus 48 fixes. They all require extensive testing to ensure that core business processes are not disrupted.
- The cost of a manufacturing defect is € 3,500 (approximately $ 4,200). This includes the cost of repair as well as the cost of downtime to the business.
- The total cost of testing, for this team testing a single SAP instance, was 1.4 million euros (approximately $ 1.7 million).
- This organization is currently undergoing a digital transformation, but the tests are 100% manual. Unsurprisingly, testing is a bottleneck to the speed of business innovation. He’s embraced agility and works in Scrums, but testing lags with every sprint. He wants to increase his output frequency, but tests are holding him back.
If you look at this business case from a purely economic perspective, the cost of testing actually goes up a bit. But at the same time, the gains in risk reduction and cost avoidance are significant: from € 2.3 million to € 19.8 million.
Delivery is expected to be nine times faster once the business moves a significant portion of manual testing to test automation (see figure below). Increasing the frequency of testing will result in nearly 10 times the number of defects detected in the tests (from 650 to 5,649).
Figure 2. After switching to test automation, the cost of testing increases slightly, but the number of defects detected increases by almost 10 times and delivery is nine times faster. Source: Jori Ramakers
With this case on how test automation aligns with the organization’s goal of accelerating the speed of innovation without compromising quality, the organization gained approval and support from top management. for its test automation initiative.
Take it to the next level
Once you have a clear view of how test automation will pay off in terms of speed, cost, and risk, you can take it to the next level. What is your business really trying to accomplish right now? Is innovation faster? Improve the efficiency of the company? Improve engagement and the digital customer experience?
Explain how your test automation initiative affects these types of high-level business goals, and you’ll likely find that it’s much easier to get management approval. Then let’s move on to the next challenge: Aligning your testing strategy with those goals, and reporting on how test automation contributes to key business initiatives.