Webinar

Breaking Barriers in Mobile Test Management

placeholder

Abstract

The mobile technology landscape is evolving rapidly, demanding high-quality apps across various platforms. Traditional testing solutions, bogged down by outdated technologies and manual processes, lead to delays and poor app performance. This session introduces a groundbreaking mobile test management approach, featuring real-device testing, AI-generated scripting, and deep performance insights. Join Erin Bailey, Director of Product Management, and Frank Moyer, CTO of Kobiton, as they cover:

  • The Magnitude of Mobile: The impact of fast-changing technology on app development.
  • Traditional Test Management Solutions: Why current practices fall short and innovation is essential.
  • Emergence of Mobile Test Management: Explore new methodologies that enhance test management and analysis.
  • Demo: Live demonstration of the latest mobile test management capabilities.
  • Sneak Peek of the Future: A look at upcoming innovations in mobile testing.

Breaking Barriers in Mobile Test Management

Discover how mobile test management can elevate your testing strategy.

Learn More

Transcript

0:00 | Erin Bailey
Hello, everyone. And welcome to our session on mobile test management. I am joined by Frank Moyer, our CTO.

0:09 | Frank Moyer
Hello, everyone. Thank you Erin.

0:11 | Erin Bailey
And I am the Director of Product Management here at Kobiton and we are going to learn about mobile test management, why Kobiton has invested to build it and also give you a demo of what we have in place as well as give you a little peek into the future. Awesome. So here is our agenda. The first part is the magnitude of mobile understanding the space. We will talk a little bit about traditional test management solutions and how mobile test management has emerged. Then I’m going to do a demo of our mobile test management solution and Frank will come back in and let you see something about the future. So, Frank, I’m going to turn this over to you.

0:57 | Frank Moyer
Wonderful. The world of testing and test management emerged in response predominantly to cross browser testing in the tens of combinations of operating systems and the browser operating systems, and the browser versions. And if you rewind about 20 years ago, for those who used Internet Explorer, and the fact that it wasn’t compliant… with the WC standard, there were a lot of testing products out there that emerged to test across different browser versions. That was the emergence of cross browser testing from BrowserStack and Saucelabs. That was their core. When you look at what’s happened in the mobile world and where we are today, it’s a whole different world. The volume of mobile is orders of magnitude hundreds of times bigger than the world of cross browser testing. And so whether you’re testing on an Lg G8 or a Pixel Eight running either browser testing or a native mobile app. The complexity that you’re dealing with in mobile is orders of magnitude greater than the cross browser testing world. So, this mobile complexity has really driven the market to move away from the traditional solutions for test management. And is the emergence of where we are today with mobile test management. I love this statistic. The top 100 mobile applications available internationally are used daily on 2,942 different device type and operating system version combinations. We have custom… who test on over 800 different device model and operating system versions combinations. And they’re running thousands of tests across those different versions and operating systems. The world. The traditional test management solutions buckle under that magnitude and aren’t designed to handle this magnitude of mobile.

3:38 | Frank Moyer
The traditional test management solutions do a great job and they do things that mobile test management are not don’t intend to do. And when you think of what traditional test management, I think of traditional test management, I think of the big four Testrail Zephyr, Qtest, and Xray, And they start with your test plan and they use your input, which are the requirements, They take those requirements. They have traceability functionality that allow you to build your test design. So your test cases and your test suites that map back to requirements. And then they allow you to execute those tests on a few browsers. So, you know, typically the problem that exists today with browsers are significantly less than… where they were 1,520 years ago because that now you have edge is replaced Internet Explorer, There is a standard that a lot of these browsers are working against. So a lot of our customers when they’re testing against cross browser testing, it’s across a few, maybe half 12 to 12 browser operating system version combinations. Then when you look at the actual defects that emerge coming out of those executions, the defects that you find on one browser is typically the defect you find on the other, It’s usually functionality within the browser application, the web application itself or the website itself, not something that’s nuanced or specific to Firefox or Opera or Chrome. And when you look at Chrome owning over 75 percent of market share, and Opera at just a few percentage points, a lot of our customers don’t even test on those different browser types like Opera. They just focus on the Firefox Safari and Chrome. So where does that lead us? And how does mobile test management fit in? If you look at mobile complexity and the ability to handle the orders of magnitude, more devices, we take the output from the test design, and we then allow you to configure that test execution across these multitude of devices. So you’ve got… hundreds to thousands of device operating system combinations. And you can now take all that those test cases and test suites and run those across a multitude of devices. And then what typically happens in the, one of the big challenges we’ve had with talking to our customers is now we find the same issue on not a 1,000 devices but maybe 50, Maybe you’ve got for example an issue with a certain screen resolution. Maybe one of our customers had a login button that was occluded on the J6 device that had a very specific resolution that occluded that login button. It happened on other devices with that screen resolution… but not other devices like the Galaxy S 20. So now you can aggregate the issues. And instead of having five different defects that are all tied to the same thing, using mobile test management and artificial intelligence that identifies what the commonalities are. So you can create one Jira Ticket or one defect and tie that back to each of the device types, saving a tremendous amount of time when you’re building out your automation. We have one customer that is required by policy to every time their test fails to create a Jira Ticket. And so they have 12 automation engineers every day who go through Jira Ticket after Jira Ticket to say, what are the commonalities? Is this truly a defect or is this a scripting issue and aggregates all that information. We’re going to be able to save them a few of those resources. They’re going to actually be able to actually write automation scripts instead of troubleshooting and finding the commonalities across different mobile device types and operating systems. The foundation of mobile test management, It also for those who don’t have a robust testing requirement, they’re just getting started. It also offers a lightweight way to get started. And what Aaron’s going to demonstrate next is that ability to get started quickly if you don’t have any of these existing mobile test management, existing test management solutions in place.

8:54 | Erin Bailey
Thank you, Frank. All right. I’m going to give a little bit more detail about Kobiton’s mobile test management solution. We’ve had test cases to support our AI powered script lists for a couple of years and we’ve had the ability to rerun those which are considered test runs, but we didn’t have it all packaged together. And so we have updated our test case functionality. I’m going to show you that We have scriptless test suites now and it’s really for having a better way to group tests and execute them all at the same time. So it’s much more efficient for testers instead of running things one at a time. And then our test run functionality… which there’s a screenshot here is pretty cool. It’s very visual to show you how your tests are performing and where you need to help the tests out a little bit and where you need to dig in and see what’s going on. So, I’m going to show you that as well. And all of this is AI powered. So that’s a big thing for us. We’re really proud of it and what we’ve been able to accomplish and hope that it really helps testers be more efficient and get things to market more quickly. Awesome. All right. So now I am going to switch screens and show you. Can you see it? Frank? Yes. And I’m going to hit play and then pause Great.

10:32 | Erin Bailey
All right. So this in case you haven’t seen Kobiton’s platform before, this is it When you first log in, you’re going to land on our device list page. A lot of you would have your dedicated devices. Those are private and local devices. Some of our customers also augment their dedicated devices with public devices. So that’s over there. And then on the far right, we’ve got virtual devices And those are currently in beta at the moment. We have Android emulators there. So we really want to offer a wide variety of ways to test all the things that Frank talked about. And then on the left hand side, just as a little overview, we’ve got sessions, This is where you can see yours and your team’s sessions. We’ve got an app repo where you can upload your apps. I’ll show you a little bit about that. And then we’ve got test management. So those are the things I’m going to talk about today. And the way we get started is really with a manual session. So I’m going to launch a device. So here is our manual session. I picked an Android device And I’m going to install an app. So this is something that’s in our app repo, It’s called Wingman. Now, I’m going to pick a version So you can see I can have multiple versions And this window stays up because I could put more apps on there if I wanted to. So I’m going to wait until it installs And it’s installed. And there it is. And I’m just going to click some things. So let’s see. I’m going to get started. I’m going to choose next and next again And, you know, I want to skip. I want to see how skip works. And skip took me right to this phone number. So, oh, I think, you know, I’m going to do email instead… Type in an email address. It’s not going to be real Don’t worry. All right. And, and I’m just going to end my test. So that’s my test. I really wanted to see what, skip how skip worked and make sure that worked as I wanted it to. All right. So after you have a manual session, you will land on in Session Explorer And this is our overview part of Session Explorer. There will be a video here. It does have to do a little bit of processing before it shows up. But the real meat of our Session Explorer is this page which I know it doesn’t look. It also has a little bit of catching up to do. Plus the first screenshot’s not there. So I just hit a play button at the bottom. It’s kind of like an iMovie like experience. You can also… drag it or tab through to see all of the steps in the test case that I just did. You can make sure that it did what you wanted it to do. And for a lot of people, this is where they see what happened in their test, They can share it with other developers. But what we’re going to do today is convert to test case which I just did. And then I’m going to go view that test case, click on that, And this takes us to mobile test management. So this is our test case details page. You can see the source session And you can always go back and look at the original source if you want. You can see that I Jane Tester created it. You can rename it.

13:45 | Frank Moyer
It’s very visual.

13:46 | Erin Bailey
It is very visual. So I’m renaming my test case And then you can see the test steps just like you did before. But let’s say that I changed my mind and I didn’t want some of the stuff at the end. I really just want this to be skipped. So I’m going to delete a couple steps Ta Da, they’re gone. And then in order for that to take effect, I do need to hit save. And once I do that, I can actually toggle between the version one and the version two, And I can decide which of those versions I want to test. This is also helpful. If you have new app versions coming down and things change, You can still test multiple app versions as well. I hope this all makes a lot of sense. And I’m going to let you look at it for just another second. And then now I’m going to show you test suite. So I have my test case. Now, what do I do with it? This is our test suites page, And you can see there’s a couple test suites already created. Each test suite is either Android or iOS because there are a lot of differences across those operating systems. And you can see that you can see when it was last run. And then if I want to create a new one, I choose my operating system, I give it a name. I’m going to give it a smoke two name. And then I’m going to pick a couple test cases and I should have renamed all of those to make it easier. But this is a demo.

15:21 | Erin Bailey

So I created a test suite Awesome Just then. So now what are we going to do? We’re going to do a test run And you can see there’s a lot of information on this page for test runs that have been done. But I’m going to create a new test run. I’m going to give it a name. We like to name things here, have custom things here. And this is for my May release. I can type… And I want to do a test suite And I’m going to pick one of the suites that I already had a login one check my version. Then I’m going to pick a device bundle and customize it. And I don’t need 19 devices. I just want to run it on a couple Some galaxies and maybe pick a pixel There we go. And that’s it That’s all you have to do. And then you would hit run. All right. So now I am going to show you something that I have done previously. Just take one second, Open up this other test run. So these are test run results that I already did with that same test suite. And as you can see, I had some success and I have some things that say need action. I did three test cases with three different devices. There’s a lot of information here. There’s some accessibility issues That’s what the little yellow dots are. There’s different things that it would take me a long time to show you everything. So I’m… to show you how I address one of the items that needs action. Cool. Here we go. So I’m going to view session detail And this is back to Session Explorer. You guys are familiar with this from a couple minutes ago And you can go through and see all the test steps. And then there’s a blocker. So I’m going to try to resolve that blocker. And if you look at this, the baseline device is a Galaxy S9 plus, and the Galaxy S9 has a slightly different screen resolution. And so it’s trying to hit the next button but it can’t get there?

17:43 | Frank Moyer
It’s occluded That’s exactly the same. Yeah. Well, I think what happens?

17:47 | Erin Bailey
Is that the blue button there is underneath it? And so it hit that other thing instead. So I have to go back a step and I have to pick that button. So I moved back a step and I picked that button and then I’m going to submit.

18:03 | Frank Moyer
Nice. So you can move back to a previous step. Wonderful.

18:07 | Erin Bailey
Yeah. So the next time I run this, it should be able to find that and so that it doesn’t go on to whatever that clipboard is, which I had never seen before Anyway. So that is the demo. Do you have anything else you wanted to see Frank or?

18:25 | Frank Moyer
Yeah, that’s really cool.

18:27 | Erin Bailey
Close on time. So now I’m going to turn it back over to you in just a minute and let me share. I’m going to reshare the other slide. Okay? All right. So, that was the demo and now let’s do a little sneak peek.

18:45 | Frank Moyer
Wonderful. I’m going to show, give it a little sneak peek of where we’re heading with mobile test management. Kobiton has an existing integration with TestRail. It’s not currently tightly coupled with mobile test management. But this is the flow that you can expect when we deliver mobile test management integration with each of TestRail, Zephyr, Qtest and X Ray. Essentially, what you’ll be able to do is start a wrapper script. So your Appium scripts will call a Kobiton endpoint, and we’ll go ahead and create that on the target system, whether it’s you know, I refer to it as Test Rail here… but it could be Zephyr Qtest or X Ray, And we will create the run within TestRail. And then you run each of your Appium scripts on different device types, and we will write those into TestRail. So when you see that test that screen that had all the different device types, and your tests running that screen will show all of our test executions that are being posted to TestRail. So now every test step in addition to being posted to Session Explorer, you’ll also see the output within TestRail. And so you can link back and forth between TestRail and Kobiton Session Explorer with a deep link. One of the greatest things about Session Explorer is you can deep link into a very specific test step and see everything that was happening. So you don’t need to go through all the steps to reproduce because it’s right there in a movie. And then when all of your script execution is done, you call the patch runs endpoint which says this test run is completed. So that’s what we will be doing from an automation perspective, integrating with the traditional test management solutions. And then I’ll give you a little view of… I think I said that one. Oh, that’s all right. I prefer this order anyway because now, I’m going to go ahead and share my screen… And let’s see. All right. So this gives you a little taste of what we’re going to be able to do from a manual perspective. And when Aaron earlier demonstrated the manual session, she was showing the device log tab. So we will be creating a new test case tab where you’ll be able to find the test case, whether it’s a test case within Kobiton or even a test case… from one of the Zephyr X Ray, Q test or test rail. In this case, it’s actually within the Kobiton system. And so you can see that we’re at this, we just finished this second test step which is validate the business card scanning with guaranteed accuracy. Yep, that looks good. And now it tells me what to do, but it’s not just text. It’s actually a visual here as well. So it says, click next, click on the next button. I can see, oh, okay. If there’s any question about what I needed to do there. Okay? And so now I can click that’s done And it gives me the next step. So now it says, you know, view, validate that cloud integration, Sure enough. And so I can go through step by step. And it guides me exactly what I need to do in executing a manual test case beyond traditionally you’re just looking at some text. And this just, we’ve gone through this with some of our customers and it’s like going to be such a more pleasant experience for them to go through versus the step by step text. It’s going to be a visual experience. And that’s one of the key things about mobile test management is it’s designed to handle the scale of mobile as well as it’s very visual because mobile is very visual… Yeah… Aaron.

23:18 | Erin Bailey
Yeah, no, that’s wonderful. I’m excited about this, Really excited about it. So thank you for giving us that peek And I think that’s about all we have.

23:31 | Frank Moyer
It looks like we have a few questions. Yeah.

23:33 | Erin Bailey
I’m going to pull up real quick while you’re answering or looking at the questions I’m just going to pull up our thank you slide. Okay. So go ahead and look at questions while I do that.

23:44 | Frank Moyer
Yeah. But the first question I think is for you. So if you want to try it out, how do you get your hands on it?

23:53 | Erin Bailey
If I want to try out, if someone wants to try out mobile test management, Oh, great. Great question. So we do have a beta program and, we just ask that you participate in some calls with us and give us your feedback for, you know, once a month for several months and we will let you try it out for free. So if you are interested, you can send an email to success at kobiton. Com, and we will work to get you set up.

24:25 | Frank Moyer
Great. And the second question is what is the purpose of the accessibility items on Session Explorer?

24:35 | Erin Bailey
I think you and I can tag team on this, but I will say one of the things I didn’t really talk about is our validations. And we do have several different types of validations that we offer for our testing and accessibility is just one of them. But accessibility is a critical component of many mobile apps that have to meet some standards to make sure that all people can use their app. And we offer a number of checks To see whether the color contrast is correct or the size of the text is correct. So that or a certain meets a certain standard, the WCAG standards, Frankie, I’m going to turn it over to you to answer.

25:17 | Frank Moyer
Yeah, that’s exactly right. And we’re we are currently support of four different accessibility validations, but we’re also in discussions with partners who specialize in mobile accessibility. We recently launched a partnership with Apple tools, which Apple tools does visual validations. So, part of our partner strategy is to partner with these companies that are, that have expertise in certain areas like accessibility or Security. We don’t want to build out any accessibility or security functionality. We’re partnering with third parties who do that really well and integrating that into our Session Explorer. Great. Well, we’re at time.

26:10 | Erin Bailey
Yes, Thank you all for joining us and we hope you have a great rest of your MTS experience.

26:17 | Frank Moyer
Great. Thank you all. Thank you Erin.

26:27 | Frank Moyer
Hello, everyone. It’s me and Erin recorded that and this is the real, Oh, and Cara. Yeah.

26:35 | Cara Suarez
Hi, We did get a bunch of questions asked live also. So one came in from Sunita around what are the best practices for maintaining test code documentation and tracking progress?

26:51 | Frank Moyer
Yeah, Thank you Sunita for the question. We’ve tried to implement as much as possible around best practices into the product. And part of it is version control and version history. So, you know, how you got to where you are today and where you’ve come from. One of the biggest challenges with test management in general is test case bloat, right? Test cases, get out of control. And you’re not sure why this test case exists to begin with, And is there duplication in your test? So you don’t Some of our customers take eight hours to run their test suites every night over hundreds of devices. And if that can be shortened because you can look at your, you have good, you are following best practices around documentation and being able to annotate and say here’s why this test case exists. Then you can clean all that up and shrink it from eight hours down to.

27:45 | Frank Moyer
six or five or four.

27:47 | Frank Moyer
So, it’s a really good point. I think there’s a lot of like insight into that question to make sure that things from Sunita to make sure things don’t get completely.

27:56 | Erin Bailey
Out of control. Yeah. And I would just add having, you know, making sure that you’re clear and concise and naming things similarly and following a pattern so that everybody knows what they’re looking at. I think it is also helpful.

28:12 | Cara Suarez
Great. We have one more short answer question we can probably do. Can you explain how the introduction of scriptless test suites in mobile test management improves the efficiency of the testing processes?

28:27 | Erin Bailey
Sure, I can start and Frank can always jump in. So there’s a couple of things I mean, first of all, the fact that you can do is create a test case from a manual session and rerun that over, you know, all types of device types and OS versions that in and of itself is such an efficiency gain. You don’t have to have automation experience, or even if you have automation experience, it might help you get to automation for a new piece of functionality more quickly. And then with test suites, instead of just having a test case, you can click a button and kick off a whole set of suites that run, you know, let’s say my test suite had three tests and three different devices. I can run those all at once and it’ll run them as it can and you can see results in real time. It’ll say running. And then all of a sudden, you’ll start to see results and you can start analyzing those as they come in. So that’s one way. Frank, do you have anything else you want to add to that one?

29:25 | Frank Moyer
No, I think that’s spot on Erin.

29:49 | Cara Suarez

Okay. Perfect. Well, if you have any additional questions for Frank and Erin or Kobiton in general, you can head over to the Kobiton booth in the expo, and we’ll have experts there waiting to answer any questions that you chat in. So thank you so much for a really exciting talk, Frank and Erin.

29:49 | Frank Moyer
Thank you, Cara. Thanks. Take care.

Ready to accelerate delivery of
your mobile apps?

Request a Demo