Bot Testing for Beginners

Remember how apps used to be? They connected to the internet, incorporated one device at most, and maybe we had to factor in a few different interrupts, but it was fine because we had great testing toolsets in place.

Fast forward to the present. Apps do so many cool things… and for that they need sensors that interface with the ecosystem around them. They sync data to your smart watch, scan visual inputs with the camera, perform biometric authentication, geo-location and, increasingly, interpret language commands.

Off-the-shelf testing toolsets have not yet caught up with the rate of change in the number of new use cases. One of the biggest trends we’ve seen is the rise of next-generation interfaces, such as chatbots and voicebots like Siri and Alexa.

Gartner predicts that by 2020 we will spend more time talking to bots than we will with our spouses and children! However you feel about that prospect it’s certainly true that bots are making their mark on digital app development. We conducted a quick poll to get a sense of how important bots are to enterprise app development and the results suggest they’re a top priority. From one-half to two-thirds of respondents were either currently using or planning to use bots in the near future. And yet the software and hardware required to facilitate today’s hyperconnected apps presents some very real hurdles for testing and test automation. In short, we need to talk about test automation for these kinds of interfaces.

1. Hyperconnected apps: a new challenge for testers

Apps that communicate with multiple sensors, and particularly chat and voicebots, require an automation strategy that accounts for their complex testing demands.

The issue enterprises are up against is that traditional toolsets are great for functional testing, but once different sensors are introduced, your usual go-to solutions require additional workarounds. Based on our poll results, we estimate those additional test runs are being configured manually for the most part. And any process that attempts to grapple with rising complexity by predominantly manual means will soon encounter difficulties.

2. Chat and voice interfaces

In the case of bots — which interpret, mimic and respond to written and spoken language — the testing challenges are as complex as language communication itself. There are factors that are common to both voice and chatbots that testers should be aware of. However, there are also challenges specific to each.

For example, intent validation is a shared test factor for both voice and chat, and as such there may be similarity in the data flows needed for both. However the demands of chat versus voice then diverge. A chat interface needs to distinguish what is meant if the query is posed as multiple questions, with typos, ambiguously or even in multiple languages. On the other hand, voicebots need to interpret different accents, tone of voice, users speaking at a distance from the sensor, background noise and the many other foibles of spoken language.  Chatbots are becoming increasingly mainstream while voice interfaces, although growing in popularity, are nevertheless a “black belt” challenge for development and test teams.

3. How to approach interface test automation

It’s clear that development and test teams require a different approach to testing the new generation of interfaces, especially in the context of hyperconnected apps and the IoT. So, how exactly should they go about it? Here we’ve shared our top five considerations for your bot test automation strategy.

4. Focus on flexibility and interoperability: The approach we’ve pioneered enables total flexibility in how tests are run and configured. We use a Mobile Automation Library which resides as part of the mobile application package, and enables different tests to be configured without resource-intensive code level changes being required. It can be controlled by a back-end interface, is modular, flexible and can scale up to accommodate more interfaces in the future. Significantly, it can be built into existing regular, automation testing for your digital initiative.

5. Automate, automate, automate: The way to beat spiraling complexity from sensors is to be smart about how and what you automate. Over-reliance on manual will ultimately drag the team down as use case complexity continues to rise. If you are unsure about how to increase the level of automation in your testing, focus first on repetitive tasks while reserving the most complex functional or flow testing for manual configuration.

6. Use quality data flows: An important piece of the puzzle in bot automation is creating data flows that mimic the interactions for your bot. Test data flows can be easily created in Excel or Mind Map, which are later “read” by the bot automation framework. However, managing the requirement changes in mind map itself can reduce maintenance time, versus converting from Excel.

7. Mix n match approaches: Headless test automation is a familiar paradigm in mobile and web circles, and can be used successfully for faster testing of bots intended for multiple platforms without tying yourself to any particular UI. By contrast, the “imitating user’s actions” approach allows enterprises to test the bot’s flow end-to-end, imitating a typical user’s interaction with the bot. This is great for testing the user experience with specific UIs.

8. Voice adds complexity: Interpreting language is a step up in difficulty, so keep in mind that voice interface automation requires additional test layers to incorporate pre-recorded speech or a Text-to-Speech (TTS) API simulator.

If you have plans to introduce bots into your digital initiative soon, or you’re currently experiencing rollout challenges, why not get in touch? We’d be happy to discuss the approaches that can work for your specific situation, as well as show you some of the tools we’ve come to rely on. And for more on this topic, be sure to listen to our recent webcast on the subject of test automation for next-generation interfaces. It’s packed with detailed content on how to make sure your bots and your customers are speaking the same language.