Mobile Test Automation Framework, Strategy & Tools

Reduce QA costs, accelerate mobile app cycle time

INTRODUCTION

The need for automation

With increasing demand for mobile services driving shorter and shorter release cycles, mobile development and Test/QA teams are under greater pressure than ever.

They need a structured method for dealing with the proliferation of operating systems and devices and their unique requirements around the world. In this paper, we look at how test automation can address these emerging needs, outline the business case for it, identify which parts of testing can be automated and suggest best practices and tools for success.

Manual testing is not enough

Software testing and QA have evolved into a whole new dimension as mobile becomes the platform of choice for customers and employees.

Mobile applications and services are in high demand and have to be developed and tested quickly and thoroughly before being pushed out to the market. A traditional QA cycle, which used to take 3-6 weeks, is now being condensed to just a few days. If the application does not meet the quality standards end-users expect, more cycles are required to remediate those issues, negatively impacting brand loyalty, revenue, competitive advantage and operating budget.

Traditional manual testing can no longer cover all of the bases

Especially when the app-to-market lifecycle has to be short. There has to be some mechanism to test an app quickly and effectively, without wasting cycles. As a result, mobile test automation framework has become a popular choice to offset these barriers and maintain adequate test coverage.

Mobile testing is challenging

A seemingly endless combinations of devices, OS’s, carriers, and networks and a lack of established best practices or industry standards around it. Mobile applications have become increasingly complex and to test the performance of the app on the server, device and network can be inundating. Existing enterprise automation tools are ill-equipped to meet the unique needs of mobile app automation testing and mobile test automation strategy requires specialists who can develop custom scripts for associated business needs.

The test automation conundrum

manual testing = big limitations

Proliferation of devices and OSs adds time and complexity

Tedious, error-prone and expensive

Non repeatability and reuse requires additional time and resources

Slows time to market considerably

Automation = lots of complexity

High cost of maintenance; very brittle

End-to-end automation requires combining strengths of different tools

Open Source tools are unreliable and unpredictable on their own

Without integration, analytics and decision support are impossible

Huge dependency on automation engineers to enhance and maintain

Key tenets of test automation

Mobile Test Automation can be complex, but it can be done effectively with the right approach. One that starts with a well thought out plan that evaluates the “what”, “when” and “how” to automate.

Find the right balance

Regression tests can be easily automated to ensure that basic functionality is maintained from the previous release. OS specific test cases can be defined and executed in multiple environments for that OS. Automation works best with features that are stable and are not undergoing changes. Not all features can be automated and require manual testing. So, there needs to be a balance between manual and automated testing.

pick the right times

QA managers must choose the right time for automation. Automating very early in the application’s lifecycle might require too many changes in the future and can significantly raise automation costs and effort. Only after manual testing has been established with repeatable processes and stable test cases, should automation be considered.

Pick the right tools

There are many mobile test automation tools in the market that provide the automation engine through a public or a private cloud platform. Some of the better known options include Perfecto , Keynote, Sauce Labs, ZapFix, Eggplant, Selenium, Robotium and Jamo. Here’s a partial list that can be used, depending on the platform. Note: This is a basic list, and new tools get added frequently.

Automation tools

Apexon’s Proprietary Test Automation Framework
For Mobile Web, Native Apps, and Hybrid Apps
Easy Integration with TestNG, Selenium, Autoit, and ReportNG

Cloud-based automated testing platform for hybrid and mobile apps
Enables users to run JavaScript unit and functional tests written with tools like Selenium and Appium
Designed for software teams of all sizes

Write and Execute Automated Acceptance Tests
For Hybrid Apps, Android and iOS Native Apps

Write Functional, System, and Acceptance Test Scenarios
For Android Native and Hybrid Applications

Open Source Cross-Platform Functional Testing Tool for SOA and Web Service Testing

Functional and Regression Test Automation for Software Applications and Environments
Mobile Test Automation Integration with Perfecto

Methodology, costs & Benefits

It is essential to take into account the overall scope and strategy of your mobile Test/QA effort in order to identify what to automate and determine the return on your investment.

Methodology

This involves identifying the right tools and technologies for automation and defining reusable test assets. Parameterizing select test cases by leveraging reusable assets to enable data driven test automation is essential. Maintaining these reusable frameworks and automated tests with changes in application is also critical.

Cost

To quantify the benefits of test automation, you must understand the true costs of automation.

It is key to begin the conversation by discussing the different cost variables involved once the automation environment has been set up. The cost of automating test processes is tangible but the net present cost and value includes many intangible factors. The best approach is to determine, with as much precision as possible, what the hard-dollar costs are today and then compare them to the hard and soft dollar benefits of automating the test efforts.

Acquisition

Test automation tools can be expensive, and are usually employed in combination with manual testing. However, over time, they become very cost-effective, particularly when used repeatedly in regression testing.

Device

This is the cost of mobile handsets required for testing. In addition to the introduction of new handsets by all manufacturers, you also have Android fragmentation, with numerous models being introduced by HTC, Samsung, Motorola, Sony and others.

Labor

Factor in the cost of trained personnel to script test cases and build manual tests. Be sure to include the costs of recruiting, hiring, paying, and retaining skilled QA engineers.

Training

Depending on which test automation product is selected, the QA team may need considerable training to become proficient at writing scripts for automated testing.

Alternatively, many companies hire professional services firms to handle the initial automated test creation. These firms may already have the expertise, tools and infrastructure in place to jump start your test automation efforts.

Benefits

In addition to dramatically reducing operating expense and accelerating time to market, test automation provides many intangible benefits as well. For example, automation can help improve employee morale and productivity because it can take on repetitive testing tasks, freeing the QA team to focus on more challenging tasks.

Discovering bugs early in the testing cycle is another intangible benefit that is difficult to measure in terms of potential impact on cycle time and customer satisfaction. The later in the development cycle defects are found, the more expensive they are. A defect identified in a production environment can be greater than 100 times more expensive to fix than the same defect identified earlier in the design phase. Automation is the key to improving the speed, accuracy, and flexibility of the software testing process, enabling companies to find and fix more defects earlier.

Detailed benefits of test automation

Reduced Testing
Costs/Efforts

Hours to create automated scripts
Hours to maintain and enhance script
Reduction in Engineer’s time requirement for test execution
Hours for 1st cycle testingHours for subsequent cycles testing when automation is available
First year person-hours requiredSubsequent year person-hours required
First year cost of testing
Subsequent cost of testing per year
Time to address defects

Reduced Time
to Production

Elapsed testing time
Automation time reduction (1st year and subsequent years)
Production rate
Total production recaptured
Margin per unit

Improved
Product Quality

Average number of defects in product reduced
Defect-finding efficiency improved
Average cost of fixing defects prior to and after release
Fixing defects pre-production
Fixing defects post-production

Intangible –
Qualitative Benefits

Increased consumer confidence
Increased ability to support the change in application functionality
Higher level of product quality
Customer satisfaction
QA team satisfaction

best practices for test automation

Types of Automation

Selecting the right test cases

Tool & Resource Capability

System/Peripheral Interaction

Interaction between multiple apps / OS-App / Multi-Domain

Location aware, peripheral, battery life, system alerts, etc.

Stability & Change

Best Practices for other GUI Automation apply

UI components that will change less

Clear understanding of business processes

Complexity & Priority

Less graphically intensive, more data and business scenarios

Medium complexity test cases:

High > Low

Smoke: High > Medium > Low

The automation architecture

Making the case for automation

Test automation can drive dramatic benefits when applied properly – reduced testing costs; faster time to market for new apps/services and faster entry into new markets and geographies.

But quantifying those benefits can be complex. In organizations where decisions are made purely on ROI, managers must keep several criteria in mind while making computations. In many cases, the intangible benefits of automation will supersede the tangible benefits. Hence, ROI computations might not reveal the total value. The best results are obtained when costs and benefits are computed in terms of opportunity costs related to test automation and manual testing. This will provide decision makers the visibility into some of the intangible costs and benefits of automation. Test Automation has worked its magic for many enterprises. It can do the same for you.

Mobile Test Automation for Faster Time-to-Market