TCP-Net and TCP-Net++ – a revolution in regression testing

In the rapidly evolving domain of software development, managing test cases efficiently and effectively has become a central challenge for quality assurance (QA) teams. Traditional regression testing methods often fall short in meeting the demands of modern, complex software projects. This gap in efficiency and effectiveness led to the development of TCP-Net, a pioneering approach using deep learning to revolutionize test case prioritization for regression testing.

TCP-Net, short for ‘Test Case Prioritization using End-to-End Deep Neural Networks’, emerged as a groundbreaking solution that introduced a novel method of prioritizing test cases by leveraging the capabilities of deep learning. As shown in Figure 1, TCP-Net analyzes coverage and history features derived from test cases and source files input features to address the growing complexity of software testing in an increasingly data-driven world. As software projects grow and become more complex, the number of test cases required for thorough testing also expands exponentially. This growth presents a dual challenge: ensuring comprehensive testing coverage while minimizing the time and resources required. TCP-Net’s innovative approach to this problem is the use of neural networks to decipher high-dimensional correlations and optimize feature extraction for more efficient test prioritization.

Figure 1. TCP-Net model inputs (Siemens EDA)

The original TCP-Net model

The original TCP-Net model represented a significant leap in test case prioritization. By utilizing an end-to-end deep neural networks architecture (Figure 2), TCP-Net could analyze vast amounts of data from software test cases, identifying patterns and learning correlations that were not readily apparent. This approach allowed for a more nuanced and effective prioritization of test cases, particularly in large software projects where the sheer volume of tests could be overwhelming.

The architecture comprises two branches in a middle fusion deep, fully connected network. Each branch serves as a feature extractor for either source-files-related or test-case-related features. Extracted features are concatenated, then fed to the main feature extractor until reaching the final sigmoid classifier layer.

Figure 2. The TCP-Net architecture (Siemens EDA – click to enlarge)

Central to TCP-Net’s methodology was its focus on coverage and history features. Coverage data provided insights into which parts of the software were exercised by each test case, while history data shed light on the past effectiveness of these test cases in identifying defects. By analyzing this information through deep learning models, TCP-Net could predict the likelihood of each test case uncovering new defects for a new build, thereby optimizing the testing process.

The initial findings from the TCP-Net study were promising. The model demonstrated a remarkable capability to streamline the testing process, reducing the time and resources required for regression testing significantly. By prioritizing test cases based on their historical effectiveness and coverage data, TCP-Net could ensure that the most critical tests were executed first, leading to earlier detection of critical defects. However, the real-world application of TCP-Net in industrial settings revealed several challenges and areas for improvement. These observations led to the development of TCP-Net++, an enhanced version of the original model aimed at addressing these challenges and further optimizing test case prioritization.

Developing TCP-Net++ and its enhancements

Building on the foundation laid by TCP-Net, the development of TCP-Net++ aimed to tackle the challenges encountered during the initial model’s deployment in industrial settings. TCP-Net++ was not just an iteration of its predecessor, but a significant enhancement, incorporating advanced machine learning techniques and addressing practical deployment issues. One of the primary enhancements in TCP-Net++ was the integration of ensemble learning, which combines predictions from multiple neural network models to enhance the accuracy and robustness of the prioritization process. Ensemble learning also helps to mitigate issues like overfitting, where a model performs well on training data but poorly on unseen data.

Another key improvement addressed the sigmoid saturation problem, a common concern in neural networks that can impede learning efficiency. TCP-Net++ tackled this issue by integrating rectified linear unit (ReLU) activation functions. These are known for their ability to maintain active gradients during training, thus facilitating better learning and convergence. TCP-Net++ also introduced the concept of ‘soft targets’. This helps reduce the sigmoid saturation problem and prevents boundary value output, further improving its generalization capabilities.

The deployment of TCP-Net++ in real-world industrial environments has yielded impressive results. The model’s enhanced capabilities led to significant improvements in regression testing efficiency. High failure coverage of 90% to 100% was achieved by running just 23% to 39% of the test cases, a notable improvement over traditional test case prioritization methods. A comparison between TCP-Net and TCP-Net++ for two Siemens EDA electronic design automation (EDA) tools is shown in table 1.

Table 1. Comparison of TCP-Net and TCP-Net++ coverage results (Siemens EDA – click to enlarge)

Implications and future directions

The development and success of TCP-Net++ have profound implications for the field of software testing. Our model showcases the potential of deep learning in addressing complex challenges in regression testing, a crucial aspect of software development. By significantly reducing the time and resources required for adequate testing, TCP-Net++ enables more efficient development cycles and quicker deployment of robust software solutions.

Looking forward, the journey of TCP-Net and TCP-Net++ sets the stage for further research and development in this area. Future work could explore the integration of additional machine learning techniques, refining the model to adapt to different types of software projects and testing environments. The potential for incorporating real-time feedback and continuous learning mechanisms also presents an exciting avenue for making TCP-Net++ even more dynamic and responsive to the evolving needs of software development.

The advancements in TCP-Net++ also open possibilities for broader applications beyond software testing. Similar deep learning approaches could be applied in other areas of software development and project management where analyzing large datasets and optimizing processes are crucial. In conclusion, the evolution from TCP-Net to TCP-Net++ marks a significant milestone in leveraging machine learning to enhance software testing efficiency. It not only presents a practical solution to current challenges, but also paves the way for innovative approaches in the software industry’s future.

Further reading

A more detailed discussion of TCP-Net and TCP-Net++ can be found in these conference papers:

TCP-Net: Test Case Prioritization using End-to-End Deep Neural Networks TCP-Net: Test Case Prioritization using End-to-End Deep Neural Networks

TCP-Net++: Test Case Prioritization Using End-to-End Deep Neural Networks – Deployment Analysis and Enhancements

About the authors

Reem ElAdawi is a test engineer director at Siemens Digital Industries Software. She has been with Siemens EDA for more than 25 years, beginning as a behavioral model engineer and moving through multiple software development and software team management positions for both analog and digital products. Her area of interest is in leveraging machine learning to enable engineers to be more effective and productive, and she has published extensively on the topic. Reem received a B.SC., M.Sc., and Ph.D. from the Electronics and Communication department of Ain Shams University. Both her M.Sc. and Ph.D. research focused on machine learning applications: speech recognition (M.Sc.), and rare event detection for high replication circuits (Ph.D.).

Mohamed Abdelkarim is a machine learning engineer and researcher at Siemens Digital Industries Software, with a background in machine learning, deep learning, computer vision, and NLP. His work at Siemens centers around leveraging innovative solutions using machine learning, computer vision, and image processing to advance the semiconductor industry, particularly in electronic design automation (EDA) tools and processes. Mohamed has also made contributions to the field through multiple international publications in machine learning and AI. He received a B.Eng. in electrical, electronic and communications engineering, and is currently pursuing an M.Sc. in AI, from Cairo University.