The AI that’s actually Intelligent: unveiling the new Kobiton NOVA AI Engine

Intro: Taking an honest look at AI

If you’ve kept up with Kobiton, you would have seen that today, December 14, 2020, we unveiled our brand new NOVA AI Engine. We (and many of our customers) are incredibly excited about it. In this blog, I want to walk through this exciting next step in AI-driven Mobile Experience. However, I want to first have a quick discussion around:

• The importance of AI in Testing
• How we built the Kobiton NOVA AI Engine and why it’s actually AI
• Everything that you can do with this “AI at the core”
  

AI in Testing: Why it matters

Simply put, AI has the potential to truly revolutionize the Testing space, especially within the realm of test automation. The industry research backs this up. This year, the latest “World Quality Report” reported that 86% of their respondents say that yes, an AI offering is now a key criterion for selecting new QA solutions. And this comes as no surprise seeing that the same report found that 88% of respondents claim that “AI is the strongest growth part of our test activities (testing with AI and testing of AI),” and that 71% reported that, “We will use more AI in our test execution activities.”

That’s a lot of steam behind the AI-in-testing revolution and for good reason. While there are many implications for AI in our space, one of the major areas where AI is being promised is within the realm of scriptless test automation. 

The flip side: is all AI just snake oil?

No, I don’t think so, or at least not all. However, there is a need to take a sober look at all of this. So much of these statistics focus on “looking ahead” into the future of AI. While I do believe the future is bright for AI, I also see the validity of the sentiment within the World Quality Report: 

“Expectations of the benefits that AI and ML can bring to quality assurance remain high, but while adoption is on the increase, and some organizations are blazing trails, there a few signs of significant general progress.”

Talking with one of our customers a few weeks ago, we were struck by something that he said when we asked about his feelings towards scriptless test automation:

“I am very interested in scriptless automation, but only if and when the AI is right. Until then, scriptless solutions just don’t give us the amount of control that we need when authoring our Test Cases. And this isn’t just my opinion. This is the broader perspective of the organization that I work in. There are a lot of tools that claim AI, but we know that from experience that it isn’t really AI. It’s half-baked.”

And he’s right. There are many “AI’s” out there being sold as snake oil. This has led to a lot of distrust across many different spaces, but I hear it a lot in the world of Testing. 

Data: the secret sauce that most tools are missing, and the reason our AI works

Now, I want to outline why our AI is legit. And why it works.

It all comes down to the data.

Writing the algorithms takes time, but it’s the data that allows the algorithms to provide business value. AI has to learn intelligent behavior “from the ground up,” and this takes information fed into the “mind” over and over and over again. And it takes a lot of information to create something that can work, “think,” and “adapt” on its own. At the heart of it, that’s really what the self-healing and continuous learning is: an AI’s learned ability (from hours and hours of crunching data) to respond to certain environmental stimuli in an intuitive manner. 

As Google’s Research Director Peter Norvig summed it up “We don’t have better algorithms. We just have more data.” (Though we think we have better algorithms too…)

This is the major blocker behind all of the brand new “AI-based tools” that we have seen hit the market, but it’s the success behind the Kobiton NOVA AI Engine. NOVA has been exploring in the background, gathering data from tens of millions of test executions, test failures, quality metrics, device specifications, element locations, NoSuchElement exceptions, etc. You name it; if there’s something that has to do with Mobile app and Mobile browser quality, you can bet that our NOVA AI Engine has gathered data and learned about it.

And the success has been incredible. But not without failures. 

The failures have been crucial

While not limited to executing scriptless test automation, this is where our AI Engine started. From the outset of collecting its first test results (think baby AI), we’ve seen maturity and accuracy improve, going from 75% to 85%, to 90%, and now close-to-perfect (Think grown-up AI in its prime). However, at those moments of 75% accuracy, that meant that 25% of the time something failed or went wrong. In most cases, this would be a bad thing. But not for NOVA. Every time that there was a failure of some sort, that failure, as well as the expected result, was data that ended up being fed into the AI mind. As the amount of data grew, so did the accuracy rate.

NOVA gets smarter with every test you run

The main reason that NOVA outperforms the other AI-based testing solutions in the market is that it was designed around the understanding that AI is imperfect and meant to learn. Too many solutions in the market expect AI to perform the magic from launch. 

With more failures, more success: this is one of the magic elements of the AI at the core of Kobiton.

So what has it learned? What’s in it for Mobile Testing and Development teams?

Now that Project NOVA has produced the NOVA AI Engine and its “out in the wild,” I want to run through, at a high level, some of the really exciting things that it’s learned how to do, and then talk a bit about where I see it going in the future. If you want an in-depth look at the new features that come with NOVA, check out our December 2020 release overview blog.

More than just testing. NOVA is an AI engine for the team focused on Mobile Experience.

Instead of building another “AI testing tool,” we decided to build a continuously self-learning AI at the core of our platform. Because of this, it isn’t limited just to testing. Instead, it will be able to act as the secret sauce behind Mobile Quality and Experience activities across the SDLC. In its current state, NOVA is able to drive:

• Scriptless automation that covers Functional and Visual Quality
• Automated Visual Validation (Text color, Element layout/structure, etc.)
• Automated Visual UX Optimization
• The ability to automatically create a fully-fledged Appium script from a captured exploratory test
• Navigate your test automation across your Device lab to ensure compatibility and usability for broad market coverage
  

For testers: comprehensive scriptless test automation

If you’ve been using Kobiton for a while, you are probably already familiar with our scriptless test automation. Our first rendition was called Intelligent Test Automation, and it made use of the first generation AI Engine that has now evolved into NOVA. So, scriptless Test Automation has existed for some time, but NOVA has made it better. 

The NOVA AI engine is the secret sauce that makes our scriptless test automation “go.” It’s the intelligent mind that is actually able to generate the test and navigate it across multiple devices while adapting to changes in elements without the test breaking. Then, if an app crashes on a device somewhere or NOVA notices a visual discrepancy or imperfection, it automatically flags the areas where these anomalies occur and point those out to you for remediation–eliminating the need to dig through logs and screenshots.

Functional Testing, Visual Testing, and (coming soon) Performance Testing

And, because it’s been gathering data all over the place, our AI isn’t limited to just Visual Testing or Functional Testing. With every NOVA-driven test that you run through Kobiton, our AI is gathering results around Functional and Visual outputs, at the same time. So, with NOVA, you’ve knocked out, not just two tests, but two different kinds of tests at once. And, we aren’t stopping here. We have plans to extend this same sort of flexibility to our new Performance Testing capabilities as well. Before too long, your testers, SDETS, Performance engineers, etc. will literally be able to do three different kinds of automated tests in the time that it takes to just perform the actual process on one device and choose a device bundle to run your test across. The potential time and effort savings here are massive.

For Testers and UI Designers

While we outlined this in more depth in another blog, I do want to briefly mention it here: NOVA can also optimize UX.

The NOVA AI Engine hasn’t just been learning how to best test the consistency of UI’s and their visual outputs. It’s also been learning exactly what makes the best and most beautiful applications in the world tick from a UI and UX perspective. Armed with that information, it was quickly able to weave-in comparisons of UI’s under test to those of the app store giants. This all happens right within the Kobiton UI. As you see the comparisons on screen, our AI is also equipped with a recommendation engine that can tell you exactly what and why it recommends a certain change in UI design.

That’s right, the NOVA AI Engine is going to tell you what beautiful is.

For SDETs and other Developers

At this point, we’ve talked a lot about scriptless test automation. However, at Kobiton, we are well aware that there are plenty of teams who really enjoy scripting. As outlined in our Intelligent Quality Suite explainer, we are never going to tell you how to automate. Ever.

We taught our AI with that principle in mind.

For the SDET who loves to script, every time that a scriptless test is created with NOVA, you can export that test as a 100% open-standard Appium script. From there, you are free to do whatever you want with it. It’s your code. Use it how you want.

We have some other Developer-centric features that we have planned for the future, like AI-assisted debugging. So, keep your eyes and ears open for future plans here!

Wrapping up: Relax. You can let our AI take it from here. 

No matter where you fit in the Mobile SDLC, your job just got a whole lot easier. Whether you’re concerned with speed and quality of releases, the flakiness of your current test scripts, or the mobile revenue generated on mobile channels, you can relax a bit. Let the NOVA Engine take the reigns, and harness the power of our AI to drive perfect Mobile Experiences for your customers. Every time.