Feb 21, 2024
Unveiling New Horizons with Industry Leaders: Our MTES Partner Panel
Adam Creamer
Read More
The AI that’s actually Intelligent: unveiling the new Kobiton NOVA AI Engine
Reading Time : 19 min read
We first unveiled NOVA in December 2020. The AI it introduced now sits at the core of Kobiton’s AI-augmented testing capabilities. Jump to “Where NOVA stands today” to see what you can use right now.
Intro: Taking an honest look at AI
If you’ve kept up with Kobiton, you would have seen that today, December 14, 2020, we unveiled our brand new NOVA AI Engine. We (and many of our customers) are incredibly excited about it. In this blog, I want to walk through this exciting next step in AI-driven Mobile Experience. However, I want to first have a quick discussion around:
- The importance of AI in Testing
- How we built the Kobiton NOVA AI Engine and why it’s actually AI
- Everything that you can do with this “AI at the core”
AI in Testing: Why it matters
Simply put, AI has the potential to truly revolutionize the Testing space, especially within the realm of test automation. The industry research backs this up. This year, the latest “World Quality Report” reported that 86% of their respondents say that yes, an AI offering is now a key criterion for selecting new QA solutions. And this comes as no surprise seeing that the same report found that 88% of respondents claim that “AI is the strongest growth part of our test activities (testing with AI and testing of AI),” and that 71% reported that, “We will use more AI in our test execution activities.”
Update: These figures come from the edition of the World Quality Report that was current when NOVA launched in late 2020.
That’s a lot of steam behind the AI-in-testing revolution and for good reason. While there are many implications for AI in our space, one of the major areas where AI is being promised is within the realm of scriptless test automation.
The flip side: is all AI just snake oil?
No, I don’t think so, or at least not all. However, there is a need to take a sober look at all of this. So much of these statistics focus on “looking ahead” into the future of AI. While I do believe the future is bright for AI, I also see the validity of the sentiment within the World Quality Report:
“Expectations of the benefits that AI and ML can bring to quality assurance remain high, but while adoption is on the increase, and some organizations are blazing trails, there a few signs of significant general progress.”
Talking with one of our customers a few weeks ago, we were struck by something that he said when we asked about his feelings towards scriptless test automation:
“I am very interested in scriptless automation, but only if and when the AI is right. Until then, scriptless solutions just don’t give us the amount of control that we need when authoring our Test Cases. And this isn’t just my opinion. This is the broader perspective of the organization that I work in. There are a lot of tools that claim AI, but we know that from experience that it isn’t really AI. It’s half-baked.”
And he’s right. There are many “AI’s” out there being sold as snake oil. This has led to a lot of distrust across many different spaces, but I hear it a lot in the world of Testing.
Data: the secret sauce that most tools are missing, and the reason our AI works
Now, I want to outline why our AI is legit. And why it works.
It all comes down to the data.
Writing the algorithms takes time, but it’s the data that allows the algorithms to provide business value. AI has to learn intelligent behavior “from the ground up,” and this takes information fed into the “mind” over and over and over again. And it takes a lot of information to create something that can work, “think,” and “adapt” on its own. At the heart of it, that’s really what the self-healing and continuous learning is: an AI’s learned ability (from hours and hours of crunching data) to respond to certain environmental stimuli in an intuitive manner.
As Google’s Research Director Peter Norvig summed it up “We don’t have better algorithms. We just have more data.” (Though we think we have better algorithms too…)
This is the major blocker behind all of the brand new “AI-based tools” that we have seen hit the market, but it’s the success behind the Kobiton NOVA AI Engine. NOVA has been exploring in the background, gathering data from tens of millions of test executions, test failures, quality metrics, device specifications, element locations, NoSuchElement exceptions, etc. You name it; if there’s something that has to do with Mobile app and Mobile browser quality, you can bet that our NOVA AI Engine has gathered data and learned about it.
And the success has been incredible. But not without failures.
The failures have been crucial
While not limited to executing scriptless test automation, this is where our AI Engine started. From the outset of collecting its first test results (think baby AI), we’ve seen maturity and accuracy improve, going from 75% to 85%, to 90%, and now close-to-perfect (Think grown-up AI in its prime). However, at those moments of 75% accuracy, that meant that 25% of the time something failed or went wrong. In most cases, this would be a bad thing. But not for NOVA. Every time that there was a failure of some sort, that failure, as well as the expected result, was data that ended up being fed into the AI mind. As the amount of data grew, so did the accuracy rate.
NOVA gets smarter with every test you run
The main reason that NOVA outperforms the other AI-based testing solutions in the market is that it was designed around the understanding that AI is imperfect and meant to learn. Too many solutions in the market expect AI to perform the magic from launch.
However, NOVA expects there will be situations when it cannot figure out the solution with confidence and will need user assistance. If NOVA hits an area of low confidence, it alerts the user to be the “human in the loop,” instead of just breaking your test. The user then performs rapid remediation. This remediation process is fed back into the AI models as data to learn from. This way, NOVA can execute and navigate more confidently going forward.
With more failures, more success: this is one of the magic elements of the AI at the core of Kobiton.
So what has it learned? What’s in it for Mobile Testing and Development teams?
Now that Project NOVA has produced the NOVA AI Engine and its “out in the wild,” I want to run through, at a high level, some of the really exciting things that it’s learned how to do, and then talk a bit about where I see it going in the future. If you want an in-depth look at the new features that come with NOVA, check out our December 2020 release overview blog.
More than just testing. NOVA is an AI engine for the team focused on Mobile Experience.
Instead of building another “AI testing tool,” we decided to build a continuously self-learning AI at the core of our platform. Because of this, it isn’t limited just to testing. Instead, it will be able to act as the secret sauce behind Mobile Quality and Experience activities across the SDLC. In its current state, NOVA is able to drive:
- Scriptless automation that covers Functional and Visual Quality
- Automated Visual Validation (Text color, Element layout/structure, etc.)
- Automated Visual UX Optimization
- The ability to automatically create a fully-fledged Appium script from a captured exploratory test
- Navigate your test automation across your Device lab to ensure compatibility and usability for broad market coverage
For testers: comprehensive scriptless test automation
If you’ve been using Kobiton for a while, you are probably already familiar with our scriptless test automation. Our first rendition was called Intelligent Test Automation, and it made use of the first generation AI Engine that has now evolved into NOVA. So, scriptless Test Automation has existed for some time, but NOVA has made it better.
The NOVA AI engine is the secret sauce that makes our scriptless test automation “go.” It’s the intelligent mind that is actually able to generate the test and navigate it across multiple devices while adapting to changes in elements without the test breaking. Then, if an app crashes on a device somewhere or NOVA notices a visual discrepancy or imperfection, it automatically flags the areas where these anomalies occur and point those out to you for remediation–eliminating the need to dig through logs and screenshots.
How NOVA works: from one manual session to every device in your bundle
Here’s what a NOVA-driven test looks like from your side of the screen.
- Test it once, by hand. Run your test case manually on a single real device in Kobiton, tapping, swiping and typing exactly as you normally would. NOVA captures every action along with the elements and screen state behind it.
- Choose your device bundle. Pick the real devices you want the test to cover, whether they are different makes, screen sizes or OS versions.
- Let NOVA run the revisits. NOVA replays your session across every device in the bundle. When an element sits somewhere different or renders differently, NOVA adapts instead of breaking the test.
- It validates as it goes. On every action, NOVA captures a screenshot and compares the visual output across devices. At the same time, it watches for crashes and functional failures.
- Step in only where you’re needed. NOVA flags anomalies on the exact step where they occurred, so you never have to dig through logs. If it reaches a step it can’t complete with confidence, it asks you to remediate instead of failing the run, and your fix becomes training data.
- Take the code if you want it. Export the test as a 100% open-standard Appium script and use it however you like.
That’s the whole loop: one manual pass in, then cross-device functional and visual results out.
Functional Testing, Visual Testing, and (coming soon) Performance Testing
And, because it’s been gathering data all over the place, our AI isn’t limited to just Visual Testing or Functional Testing. With every NOVA-driven test that you run through Kobiton, our AI is gathering results around Functional and Visual outputs, at the same time. So, with NOVA, you’ve knocked out, not just two tests, but two different kinds of tests at once. And, we aren’t stopping here. We have plans to extend this same sort of flexibility to our new Performance Testing capabilities as well. Before too long, your testers, SDETS, Performance engineers, etc. will literally be able to do three different kinds of automated tests in the time that it takes to just perform the actual process on one device and choose a device bundle to run your test across. The potential time and effort savings here are massive.
Update: Performance is no longer “coming soon.” Kobiton now offers baked-in performance assertions, so you can catch performance issues in the same runs that cover functional and visual quality.
What makes NOVA’s visual validation different is that it was built for mobile from the start. Most visual testing tools started on desktop web and later stretched to mobile. NOVA learned on real mobile devices, and that shows up in a few practical ways:
- You don’t capture baselines. NOVA takes a screenshot with every test action, infers the baseline and compares visual output across your devices for you.
- Fewer false positives. Because NOVA runs on real devices, it can read the app’s actual element structure, not just pixels, so it is better at telling a real defect from harmless rendering noise.
- Visual and functional results from the same run. You don’t need a separate visual pass, because the validation happens in lock-step with functional execution.
Here’s what one NOVA-driven session covers compared with doing it the traditional way:
| Test type | The traditional way | With NOVA |
|---|---|---|
| Manual functional test | A tester performs it on one device | You still do this once. It’s the input. |
| Cross-device functional regression | Write and maintain scripts for every device | NOVA replays your session across the whole bundle |
| Visual validation | Capture baselines and configure comparisons | Screenshots on every action, with inferred baselines compared across devices |
| Crash detection | Dig through logs device by device | Crashes are flagged on the step where they happened |
For the full story on how NOVA approaches visual testing, read Get ready to Pioneer the next era of Visual Testing with the NOVA AI Engine.
For Testers and UI Designers
While we outlined this in more depth in another blog, I do want to briefly mention it here: NOVA can also optimize UX.
The NOVA AI Engine hasn’t just been learning how to best test the consistency of UI’s and their visual outputs. It’s also been learning exactly what makes the best and most beautiful applications in the world tick from a UI and UX perspective. Armed with that information, it was quickly able to weave-in comparisons of UI’s under test to those of the app store giants. This all happens right within the Kobiton UI. As you see the comparisons on screen, our AI is also equipped with a recommendation engine that can tell you exactly what and why it recommends a certain change in UI design.
It helps to separate the two things NOVA does with your UI, because they answer different questions:
| Visual validation | Visual UX testing | |
|---|---|---|
| The question it answers | Does my UI look the way it should on every device? | Is the way it should look actually good? |
| What it compares against | Your own baseline, across devices and OS versions | Design standards learned from the app store’s leading apps |
| What you get back | Flagged discrepancies to fix | Recommendations, with the reasoning behind each one |
| Who acts on it | Testers and developers | UI and UX designers |
That’s right, the NOVA AI Engine is going to tell you what beautiful is.
For SDETs and other Developers
At this point, we’ve talked a lot about scriptless test automation. However, at Kobiton, we are well aware that there are plenty of teams who really enjoy scripting. As outlined in our Intelligent Quality Suite explainer, we are never going to tell you how to automate. Ever.
We taught our AI with that principle in mind.
For the SDET who loves to script, every time that a scriptless test is created with NOVA, you can export that test as a 100% open-standard Appium script. From there, you are free to do whatever you want with it. It’s your code. Use it how you want.
Scriptless, generated Appium, or fully scripted: which fits?
We’ll never tell you how to automate, but we can tell you where each approach tends to shine:
| Your situation | Best fit | Why |
|---|---|---|
| Manual testers need to automate regression without writing code | NOVA scriptless automation | Their manual session becomes the automated test across every device in the bundle |
| An automation engineer is starting a new test | Generate an Appium script from a manual session, then refine it | No blank file. The navigation is already written, and they add the logic |
| A release candidate needs a broad compatibility check, fast | Scriptless runs across a large device bundle | Maximum device coverage for one manual pass |
| The flow needs complex conditional logic, test data setup or backend calls | Fully scripted Appium in your own framework | Full control over what a recorded session can’t express |
| A mature Appium suite lives in CI | Keep scripting, and add self-healing execution | Fewer flaky failures without rewriting anything |
Most teams end up using all three. That’s the point: the AI underneath is the same, whichever way you choose to work.
We have some other Developer-centric features that we have planned for the future, like AI-assisted debugging. So, keep your eyes and ears open for future plans here!
Update: Since this post, we’ve also added self-healing script execution for Appium. When an element locator changes, your scripts keep running instead of throwing a NoSuchElement exception.
Where NOVA stands today
When we launched NOVA, we said it was built to learn from every test you run. It has kept doing exactly that. At launch, NOVA had learned from tens of millions of test executions. The AI behind Kobiton’s testing platform is now trained on over 80 million tests, and the capabilities this post introduced have grown into a full set of AI-augmented testing features.
| Capability | What it does today | Where it started in this post |
|---|---|---|
| No-code (scriptless) test automation | Perform a manual test on one real device, then run that same test across your other real devices automatically. | “For testers: comprehensive scriptless test automation” |
| Appium script generation | Generate an open-standard Appium script from any manual session, not only from scriptless runs. | “For SDETs and other Developers” |
| Self-healing script execution | Keeps scripts running when element locators change, cutting down on NoSuchElement exceptions. | The self-healing described in “Data: the secret sauce that most tools are missing, and the reason our AI works” |
| Visual testing | Detects visual anomalies automatically as your test runs across many real devices. | “Functional Testing, Visual Testing, and (coming soon) Performance Testing” |
| Automated crash detection | Replays your test across many devices to surface crash scenarios. | The crash flagging described under scriptless automation |
| Performance assertions | Adds performance checks to your test runs, so you no longer need a separate performance pass. | Listed as “coming soon” in this post |
The principle hasn’t changed either. NOVA still expects to be imperfect, still asks a human to step in when its confidence is low, and still learns from every correction.
NOVA AI Engine: frequently asked questions
What is the NOVA AI Engine?
NOVA is the AI engine at the core of the Kobiton mobile testing platform, first released in December 2020. It learned from millions of real test executions, failures and device specifications, and it drives scriptless test automation, automated visual validation and Appium script generation on real mobile devices.
Do I need to write code to use NOVA?
No. You run your test manually on one real device, and NOVA turns that session into an automated test across the other devices you choose. If you do want code, you can export the test as an open-standard Appium script.
Can NOVA generate Appium scripts?
Yes. Every scriptless test can be exported as a 100% open-standard Appium script. Since April 2021, you can also generate a script from a manual session without running scriptless automation first.
What happens when NOVA isn’t sure what to do?
NOVA is designed to know when it’s uncertain. When it reaches a step it can’t complete with confidence, it alerts you to step in rather than failing the test. Your correction is fed back into the model, so NOVA handles similar situations more confidently next time.
What kinds of apps can NOVA test?
NOVA works with native and hybrid mobile apps, and since the December 2020 release, scriptless automation also runs on mobile browsers. All testing happens on real devices, not emulators.
Does NOVA replace manual testers?
No. NOVA multiplies what a tester can do. The tester’s manual session is the input NOVA learns from and scales across devices, and the tester remains the human in the loop whenever NOVA needs a decision.
Wrapping up: Relax. You can let our AI take it from here.
No matter where you fit in the Mobile SDLC, your job just got a whole lot easier. Whether you’re concerned with speed and quality of releases, the flakiness of your current test scripts, or the mobile revenue generated on mobile channels, you can relax a bit. Let the NOVA Engine take the reigns, and harness the power of our AI to drive perfect Mobile Experiences for your customers. Every time.
Want to see NOVA in more depth? Start here:
- Get ready to Pioneer the next era of Visual Testing with the NOVA AI Engine: how NOVA approaches visual validation and Visual UX testing on mobile.
- Comprehensive mobile quality gets a whole new name: The Kobiton Intelligent Quality Suite: where NOVA fits in the wider platform.
- AI Augmented Testing: everything Kobiton’s AI does for your team today.
- Scriptless Test Automation: turn manual sessions into cross-device automation.
- Appium Script Generation: get open-standard Appium code from any manual session.
