UserTesting bolstering ML-driven UX testing with $1.3B acquisition by Sunstone, Thoma Bravo

Gaining insights into how users experience and make use of software used to only be possible by having humans do all the user testing. With the advent of modern sentiment analysis and machine learning (ML) techniques, more insight than ever before can be gained from testing.

UserTesting is among the pioneers in the space using ML techniques to help discover and analyze user behaviors. The past two years have been a whirlwind of activity for the company. In 2020, UserTesting raised $100 million in funding, and a year later in 2021 the company went public on the New York Stock Exchange (NYSE) under the symbol USER.

Today, UserTesting announced that it has entered into an agreement to be acquired for $1.3 billion by Thoma Bravo and Sunstone Partners. When the deal closes, the plan is to merge UserZoom — which Thoma Bravo acquired in April 2022 — with UserTesting, to create an even larger set of capabilities for user experience testing.

“We are in a space where we’ve built a set of technologies for capturing a kind of feedback we call a customer experience narrative,” Andy MacMillan, CEO of UserTesting, told VentureBeat. “UserZoom has a set of additional different techniques and research methodologies that could supplement some of our customer experience narratives.”

How UserTesting integrated ML

Over the last two years, UserTesting has made significant investments in technology that help it distill insights from its testing.

The testing involves recording users to see how they interact with applications, including what they click on, and having users narrate their experience. MacMillan said that his company invested in using ML to help extract insight out of the recorded user experience content.

“We’re really taking unstructured content, but turning it into something structured,” MacMillan said. “We’ve trained a set of machine learning models to help discover what we call the moments of insight.”

The moments of insight are those nuggets of information that can help identify trends that will improve user experience. UserTesting makes use of multiple ML technologies, including natural language processing (NLP), computer vision and intent and behavioral analysis.

Among the things that ML enables for UserTesting is the ability to do clickpath analysis, which can track where a user goes and what they are actually trying to do when clicking something. User-sentiment analysis is another key attribute that ML helps with, as well as the ability to see if the user is satisfied with an experience. 

Going a step further, UserTesting uses ML to help power a visualization that overlays intent and path behavior to get insight into how people go through a site or an application.

“There’s a bunch of things that we can determine about the behaviors that we’re seeing people exhibit, while they go through a process,” he said.

The virtuous cycle of ML 

ML doesn’t exist in a vacuum; by definition it’s about machines learning from data.

MacMillan explained that the UserTesting approach to ML is a virtuous cycle, where the models that his company builds are continuously validated and expanded with new data from user-testing sessions that already benefit from ML. He added that the ability for humans to validate ML models with their own eyes helps build confidence in the models.

“We collect these customer experience scenarios — sort of end-to-end videos — and we use the machine learning models to point people to the moments of insight,” MacMillan said. “But you can always dig in, you can always say ‘oh the model says this, let me watch part of this customer experience narrative,’ and see if the intent really matches the sentiment.”

One of the biggest challenges overall with ML for any organization, in MacMillan’s view, is having the right kind of training data. UserTesting already has video capture, which shows what’s happening on a screen, and the test also collects click data from the users. The tests are conducted against a test plan, so there is a baseline expectation for what users are supposed to do. UserTesting has dedicated staff that are also labeling content as part of their day jobs to help train and optimize the models. 

“The point of the product is to help connect teams directly to real customers and real human beings to get human insight out of the product,” MacMillan said. “We think machine learning is really just a vehicle to help people connect to those moments of insight, but those moments are still human.”