Smart Regression Testing within a CI/CD Pipeline

With the benefits of agile practices being publicized more and more each day, many companies are adopting agile approaches for software development. This is allowing companies to roll out new features to their software at an accelerated pace, and sometimes even daily. However, regression testing plays a vital role in ensuring that the product remains operational when a new feature is added with each software update. As each change carries a risk of introducing new regressions in the next release, frequent regression testing should be done to ensure that the bugs are caught quickly.

The importance of smart regression testing in CI/CD

Let’s take this common example: An application has to execute 300 test cases. We can assume a 30 test cases per module setup that takes approximately 3 hours for the entire regression suite to complete. During the execution of a CI/CD pipeline, if any changes are made in a particular module, then in the next cycle/run it executes the entire regression suite again.

This leads to an unnecessary wastage of time. Additionally, it is a challenge for any developer to see the benefits as they must wait due to incremental changes in between regression cycles. To overcome this situation and execution anomalies, many companies follow these two approaches:

• Data-driven regression testing: requires CSV, excel, or any kind of manifest file to execute regression test.
• Input-driven regression testing: based on a manual pattern analysis, users select the required test suit and act accordingly while providing inputs.

Do you feel there is something wrong?

When the numbers of changes increase then these processes are successful, but it does not provide efficiency. Instead it becomes an exhaustive task as there is no significant decrease in terms of effort, resource, or time. This is where ‘smart’ regression can help to reduce execution time, cost and throughput of the system by executing only test cases from that module (i.e. 30 + updated test cases).

How smart regression testing as a part of CI/CD can make companies more efficient

CI pipelines, as part of smart test case execution will analyze the code commit data and based on that analysis it will execute the required test cases. If we follow the approach of smart regression testing, we can run the testing on the basis of provided labels in an issue tracker. This can ultimately save human effort, resources and helps in faster delivery with more consumer satisfaction and overall outcome.

Smart regression testing in a CI/CD pipeline will perform the following tasks to execute test cases smartly:

• Check-out code from the repository.
• Download project dependencies and build the code.
• Analyze a commit message of code and find out respective issue IDs of Issue tracker system (Jira) as per the specified pattern.
• Based on the Issue IDs, it will gather all the labels of an issue (labels are provided to issues based on the component/module or user-story) and prepare a list of unique labels from multiple issues.
• It will compare the collected list with a pre-defined list of valid labels for data correctness. An exception will be raised if any invalid label is found.
• After that, it will invoke the testing framework/execution engine with a list of labels as arguments.
• Based on labels, a test case framework will execute the specified test groups or test suites only.

Smart regression testing in a CI/CD pipeline can be used to make releases lean, efficient and intelligent because it helps us to reduce the test execution time and exclude unnecessary test cases. Submit the form below today to learn how we can help you!