First Things First
To use valuable resources efficiently, complex software regression tests call for intelligent prioritization. Those tests should be carried out first which stand a high chance of finding errors and test critical areas. For humans, prioritizing important test cases is a difficult job to do, whereas machine learning delivers the best sequence. This is why IAV uses this technology in areas such as managing software tests, in customer projects.
The “First things first” maxim is the fastest way to make progress in testing software systems. Following module and integration testing at the end of the development process, checks are carried out to make sure that all components in the entire system interact as the customer specifies. This involves taking many requirements into account, with changes at one point often leading to problems in other areas. This makes a huge job of testing and planning the input involved and has to be repeated with each new software version.
Testers often face a black-box
“Often, there are several thousand test cases that would have to be worked through manually in regression tests. On top of this, testers in most cases face a black-box in system tests. The source code usually comes from different component suppliers and is not known, either for confidentiality reasons or because they use third-party libraries,” reports Dr. Remo Lachmann, head of the Software Testing team at IAV. “This means it is not possible to analyze code changes to prioritize test cases in advance and identify potential sources of error early on.”
In addition, there is often an insufficient number of human testers for this demanding task. Plus the fact, they are usually under great time pressure which calls for maximum efficiency. “Sometimes three testers need to carry out more than a thousand test cases in just two weeks before a software release – an impossible task,” Lachmann says. Although test automation can help to reduce manual testing effort, it often comes with high initial and updating effort which, in most cases, rules out all-embracing test automation. “The only way to optimize the use of available resources consists in prioritization.” For humans, however, this is a difficult exercise in many test cases.
But there is an alternative. Using an innovative method from IAV, which is based on machine learning, test cases can be priorititized automatically according to importance and, therefore, processed more effectively.
First, an expert trains a model which can then automatically prioritize any chosen test cases. It shows testers what is important and what’s not. “Humans don’t scale half as well as computers in this field,” Lachmann says. “Studies have shown that we human beings can put no more than 100 things in an objective order of sequence.”
Significantly higher fault detection rate
IAV’s system takes care of prioritizing. Once the most important tests have been completed and the time allocated for testing has elapsed, testers can finish their work and still be sure that they have done their best to provide their customers with fault-free software. “Our practical experience shows that our prioritization tool significantly increases the fault detection rate,” reports Lachmann. “Faults are found earlier and the efficiency of software tests increases.”
IAV customers can benefit from the system’s efficiency and performance. “We offer software test management as a service and use machine learning to prioritize test tasks,” Lachmann says. “Our service includes analyzing existing test cases, test data and, if necessary, providing suggestions for improving them. In addition, we analyze and optimize our customers’ existing test strategies. As customers, Lachmann not only focuses on automotive companies. Every software company can use the service.
There is no limit to the spectrum of programs that can be tested. It ranges from embedded software, apps and desktop applications to back-end solutions. “We can coordinate and manage tests and, if the customer so requests, also the process of actual implementing and executing them,” Lachmann explains.
Individually tailored service
Exactly what this service needs to cover can be agreed on a case-to-case basis. If a customer is confronted with an acute problem, IAV’s experts can provide assistance at short notice. Support for a complex test project can take several months. Lachmann and his colleagues can take care of the entire test management process, not only including organization but also the interface between developers and testers. “Besides our broad experience in this field, using machine learning in system testing is our unique selling point,” Lachmann says.
The article was published in automotion 02/2019, the automotive engineering magazine of IAV. Here you can order the autmotion free of charge.
