Assuring Chatbots Deliver the Right Customer Experience - Continuous Testing is Central

Since they were first introduced nearly a decade ago, the development of chatbots has made enormous strides. In May this year, a Chinese team of scientists introduced us to the “emotional chatting machine” signalling the approach of an era in which human-robot interactions are seamless and go beyond the purely functional. The ECM, as it is known for short, is able to produce factually coherent answers whilst also imbuing its conversation with emotions such as happiness, sadness or disgust.

Seen by many of as “an important step” towards personal applications that can provide a viable alternative to human assistance, chatbots open the door for implementing new business models or reviving old ones. The pressure is on for organisations to innovate faster.

But - early adopters are learning an old truth – innovation must meet user expectations of quality.

Bot controversy highlights quality issues

According to analyst firm Botanalytics, while a projected 35.6 million people will use voice-activated assistants at least once a month this year, 40 per cent of bot users disengage after just one interaction. This means the stakes are high for the many brands and developers delivering audio experiences.

And, whilst there are many examples of good uses of this technology, there are also enough bad ones to demonstrate the risks of not taking quality seriously enough.

In March this year, Microsoft launched a chatbot named ‘Tay’ designed to have conversations with Twitter users, and learn how to mimic a human by copying their speech patterns. It was supposed to engage with people aged 18–24 but users of the notorious 4chan forum instead taught her to tweet offensive phrases and left Microsoft with a tricky damage limitation exercise. And Facebook faced similar problems earlier this year, when it was forced to shut down a pair of chatbots in the social network’s artificial intelligence division after discovering that they had created a secret language all on their own.

Of course, these stories play on a deeper fear for some -  a paranoia of "computers replacing people in the workforce" and "technology [they] don't understand. But for us, the only way to address these fears is to assure the quality of service needed to captivate, delight - and crucially -- help -- users accomplish what they want, when they want and how they want.

Simplicity is your friend

In technology quality assurance is never easy– but for many, simplicity is key. While Cortana, Siri and Alexa might eventually develop “‘ask me anything” capabilities, without unlimited budgets and huge development teams, for enterprises, it’s better to deploy task specific bots to engage your audience. The key is to set clear goals and identify high value or high frequency use cases for your chatbot. Don’t attempt to address problems beyond your scope. Instead, manage customer expectations by keeping the conversation within clear parameters. For example, this means responding to queries such as, “what are current interest rates” before answering open ended queries such as “what is the best auto policy for my family.”

But, of course, however simple your objective is, like any other emerging technology, chatbots inevitably add complexity to applications. Adding capabilities through this new channel, together with the difficulty of processing open-ended conversations, means that the development of this type of application can be extremely complicated. Development teams need to strategise on bespoke solution integrating an AI engine (like IBM Watson), together with a Natural Language Processing speech engine (such as Microsoft Bot Framework) to power these capabilities or leverage APIs to connect to various specialist bots. But the bottom line is that they must work hard to create a platform where systems of engagement and systems of record can work together to power a seamless customer experience.

The importance of continuous testing

For development teams trying to deliver innovation faster and leveraging agile and DevOps practices, testing is the answer to quality assurance. But, as any developer burned by late night debugging appreciates a robust testing suite, and its ability to ensure well-functioning code and developer sanity - many are turning to continuous testing to iterate faster, cheaper, and with confidence that they’re not introducing new bugs as capabilities are exposed to users.

Chatbots bring their own set of challenges to testing. Voice activated technology and chatbots need to accommodate a broad dictionary, and in the future, add imagery inputs. Languages, voice variation, accents and speech impediments all need to be catered for including the usage of “millennial slang” adds another level of complexity to a burgeoning industry. And, of course, assembling a test environment that resembles real user conditions along with - integrating third party platforms, like Facebook Messenger or Telegram - is hard

For many teams, initial testing efforts are manual – it’s not an oversimplification to imagine several engineers in a room talking to “Kate” using their smartphone, inquiring about insurance options or trying to find out the balance of their current bank account. But, to keep up with the speed of digital, test automation is the critical enabler. Moving voice and chatbot testing from lengthy manual processes to an automated approach is the proven way to increase the effectiveness, efficiency and coverage of testing. And having an automated solution for testing means you’ll test more often.

Furthermore, embedding testing throughout the development process means that uncertainty is reduced and feedback loops are accelerated, ensuring that development teams’ assumptions are correct and that they’re building in the right direction. With chatbot technology, just like any other new capability, the earlier quality issues are dealt with; the less focus is taken from teams’ primary goal of building great services.

Agile development methods are also a crucial component in ensuring speed – and in chatbot technology, velocity is critical to introducing the new innovative capabilities which help enterprises stay one step ahead of competition. The need for speed means that agile development, and automated processes, is a must have rather than a nice to have.  Manual testing means slowing down feedback to developers after an issue has been introduced – and makes the development process cumbersome and unnecessarily extended.

An expansive future - with firm foundations

There’s little doubt that voice technology is going to fundamentally change the way users interact with applications. Chatbot deployment might be just one element of the wider pursuit of better digital engagement, but it’s a crucial tactic – and one which may well be a key component in determining the ultimate digital winners.

The world’s largest organisations are rarely wrong, and as Facebook, Microsoft, Amazon, and others are invest heavily in implementing chatbots, it’s no surprise that others are quickly following.

But, for us, quality is paramount, and the companies who get it right be those who start from firm foundations – investing in agile development strengthened with DevOps practices, enabled by continuous testing to put the customer experience first – right at the heart of innovation.