Katalon: Revolutionizing Testing with Generative AI

Video Transcript

0:00 | Alex Martins
Hey, everyone. I’m Alex Martins and I’m the VP of Strategy here at Katalon, I’m really excited to be here today to talk to you about how generative AI is revolutionizing the testing industry. Move. All right. So, what should you expect in this session? We’ll cover a quick journey of AI and software testing. We’ll talk about practical implementation examples of AI. We’ll discuss some lessons learned that we here at Catalan have gone through over the past year and specifically in different AI driven use cases that we have experimented and implemented. We’ll talk about data privacy and how that is critical as we integrate AI into the different products and applications. We’ll talk about AI as a tool, not necessarily as a replacement. And then we’ll start looking forward. We’ll look into what we are expecting.

0:59 | Alex Martins
To see around AI advancements in 20 24. And then we’ll open up for some QA. All right, let’s do this. And before we start, I just wanted to share with you this slide because I think it’s important to, for you to understand that everything that we’ll be sharing today has been informed by just working with our over or almost 2000 paid customers around the world across the most diverse industries from the more modern forward thinking technology companies, to the more traditional slower to adapt new technologies such as government agencies, for example. So everything that we’ll be sharing has been learned as we interacted very closely with many of these organizations. All right. So in terms of the journey that we’ve seen that we’ve embarked a little over… a year ago with generative AI, to be more specific, we see that there is this shift in perspectives, right? Whereas in the old days, not so many years ago, the focus was really on analyzing huge amount of data and then from that data try to predict new events, right? So essentially analyzing massive amounts of data, implementing different AI, machine learning techniques like clustering optimization, and so on. And based on past performance or past events, predict what will likely happen in the future. So that’s the more traditional side of AI that we’ve seen being used very effectively across different software testing solutions. Today. Where we are, is, we are seeing this shift from that more reactive… proactive approach to applying AI to a more proactive approach with generative AI. We’re not yet in that fully proactive because the technology around generative AI is advancing daily. I mean, we just had the latest version of GPT Four being unveiled which is pretty impressive. And I’m sure more of these types of news are going to be coming out faster than it is today. But we’re in this midst of shifting from that reactive approach where we analyze past data and try to predict what things will look like in the future provided the parameters stay close to what they were to this more proactive approach where we don’t have to necessarily analyze past data in order to start generating new data. And then tomorrow where we’re seeing things going is this, you know, everybody’s sort of embracing this state of hyper productivity to really start tackling complex challenges that we can’t that we’re unable to tackle today. So there’s a lot of expectation around unlocking productivity gains like never before seen powered by generative AI technology. We are not there yet. A lot of people think we’re there yet We’re there already, But we see a lot of still unanswered questions and lots of gaps quite frankly in the technology as it is today. But like I said, it keeps on improving at impressive rates. So I won’t be surprised if in the next release of the different large language models, we see some… completely revolutionary state But that’s kind of where we see we’re going from this more traditional AI machine learning world to this more predictive proactive use of AI to really increase the productivity of the different teams, different individuals like never before seen. And in terms of practical implementation. And the reason I like this slide when I put it together is because last year we’ve been hearing and we’ve been talking about theoretical use cases, right? Or theoretical application cases of AI like, oh, it would be great if we could apply AI here and the potential is huge. So let’s give it a try And then many of us try and then we see mixed results. So there was a lot of experimentation last year… And you know, quick example here in this image that, you know, hey here’s my coding task beautiful Mona Lisa. And then here’s my sketch of what the actual code comes out. There was a lot of experimentation. And because of that experimentation, there has been a lot of evolution, a lot of learnings through evolution. And we are seeing this movement now this year Where we’re moving beyond the realm of possibilities and we’re starting to see real practical results. We’re seeing that across different professions, We’re seeing that across different disciplines in the software development lifecycle. And the bottom line that we see is that In none of those use cases, AI seems to be coming as a replacement to any of the professions. It seems that at least for now… it’s more of a collaboration tool to augment the team’s productivity or to augment the team’s ability to tackle complex problems that used to take a ton of time. If they were to do it by themselves in the past, A clear example us here at Katalon as a software tool vendor, we are using AI ourselves as we develop our products here at Katalon And the productivity gains by just leveraging generative AI in our development process. Ai has been incredible. So if you extrapolate that to organizations that are building e commerce applications that are building mobile apps for banking, financial services, et cetera… infusing AI into those applications. Is something that we’re seeing happen a lot more as organizations… are becoming more comfortable with AI and working with AI… Excuse me, And throughout, you know, all this time that we have been working with generative AI and experimenting and all. We’ve had lots of different lessons learned and I like to share those lessons learned specifically around different use cases that have been driven by AI. So the first one here we’re seeing that as we worked with different developers in our customer base, it’s been in different testers. It’s been very clear to us that the way the developer look at AI is very different from the way that the tester looks at AI. Developers seem to be much more open to working with AI in order to help him or her in a… way of accomplishing their tasks. The role of AI for developers seems to be more revolving more around providing different kinds of solutions for things they have to do for tasks that they have to accomplish, be it, you know, write code to connect to an S3 bucket and do whatever, or get an authentication token in whatever multi factor authentication scheme. Ai helps do those things, providing a lot more speed in how the developer would do it by themselves. So for a developer, the bottom line that we’ve observed and learned last year is the use cases for them is if AI can let’s say in the code generation use case, if AI can generate code for them, that accomplishes what they’re trying to do… that’s good enough. They’ll run the code, If it does what they were expecting, they’ll move on, right? They’ll check in their code, submit a PR and go on and move on to the next task. In their backlog. Testers, however look at it differently. So for them because their job is different, their job is really to ensure that a certain feature or functionality not only are working but are performing are secure, right? So the job is not so much to create something and then see if it’s working. It’s more somebody else created the application feature. For example, Now, I have to use my judgment, my knowledge, my SME knowledge to judge if this is working as per the requirements as a user would expect… if it’s performing and so on. And so forth. So just giving, so to talk in practical terms. So to read requirements, right? Give requirements to an AI solution and tell AI to generate tests to validate that the features specified in the requirements. Hmm, doesn’t seem to give the lift that testers would expect Because the first thing they do when they receive those tests that were generated by AI, what do you think it is? They’re testers, right? So something generated those tests. The first thing they do is they want to see if those tests are right, right? And how do you do that? I mean my SME knowledge is different than your SME knowledge from his SME knowledge. But if the three of us look at the same output of a system, we will have different perspectives… And a tester will, you know, look at that output and the tester will want to get comfortable. The tester will review the test that was generated, will want to say, okay, why did the AI think this test had to be created? This way? We’ll look at the requirements. We’ll try not to reverse engineer And obviously, with generative AI that’s not possible. So, we’ve seen again and again throughout last year testers trying to do that. And what happened actually is the productivity of the tester that was trying to do that went down because now they were not only testing the application they need to test, but they were also testing the output of the LLMs. And so, It’s really interesting how the mindset of a tester works in a different way than the mindset of a developer. And that example… that use case has made that crystal clear. So, the point here or the lesson that we learned here is that we need to think about AI powered use cases for testers in a different way than what we think than when we think about a use case for a developer. Just the use cases are not, you know, they don’t convert well from a developer to a tester. We need to think about a different experience for the tester as they interact with AI. Then once we get past that, we learn also that we need, as part of that process, we need to help the tester understand how much gains they are achieving in terms of increasing their efficiency, their productivity, and so on and so forth. By using AI, right? There has to be data to convince the test… and their management that an AI powered tool or feature is really giving them the benefit that justifies them to use that, right? So, and that’s very hard to do in a generative AI setting because, you know, generative AI’s output are not, you know, deterministic, they’re not repeatable, right? They’re not explainable and so on and so forth, right? So, how do you help the tester and their management understand the essentially the ROI there, right? Is it really giving them any value and benefit that justifies them to move forward with embracing AI as part of their day to day And then where we see things going in the future based on all those lessons that we have learned, is that we, it’s very clear that generative AI needs to power testers… as well? It’s not very clear or it’s not completely clear that. Specific use cases that will allow that to happen. There are many different experiments that we ourselves are running. We see, you know, others in the industry running and we’re trying to converge to a use case that not only we think is delivering the value, but also we can show hard data. And more importantly, we see our users embracing it. So, very good traction. So far in the use cases we’ve put out. But there’s a lot more to polish and learn and continue to learn by working directly with our customer base. So it’s been a very interesting and exciting learning journey over the past year or so. And in that context, as part of embracing AI, obviously.

15:56 | Alex Martins
data privacy has been another huge learning for us in the sense that what do we need for AI to actually work? We need data. So the more data you expose to AI, the more quality output you get. But then obviously companies start getting concerned, They don’t want to share IP, They don’t want to share PII of their users, of their end users and so on and so forth. So all absolutely valid concerns. So how do we address those? We really need to work with the different AI and security organizations or teams? We see, most companies that we’re working with are literally just putting together their AI governance model. And so they are working with their vendors to really shape that, right? Because nobody knows exactly the capabilities… and the limits of what AI can do. So, we see this the AI committees if you will in the company that we’re working with, and they’re working with all their vendors and collaborating to really build out the policies, that not only protect the company itself but also give them the flexibility to leverage the benefits of AI Because you can’t lock it down but at the same time you can’t open everything up. So what is the right middle ground there to be achieved? Obviously, that has to be balanced with performance of the different large language models. If you want to host everything on prem, there’s a cost to that. There is… a time and effort commitment that the organization has to dedicate because hosting your own large language model behind your firewall on prem, that means you’re going to have to fine tune it to train it. And that’s a very expensive process depending on how fancy you want to get. And then if you want all vendors to use that on prem, large language model, then that essentially means that you’re going to have to work with all the vendors to make sure that the output of your large language model right inside your firewalls are providing are performing and are providing quality output to the vendor solutions, right? Because vendors, you know, like ourselves, for example, we need to also make sure that what AI, the way AI is integrating into our solution is not hindering the ability of our solution to… give our users high quality data. So it’s a very interesting time we’re in right now as we work with our different customers throughout those pilot projects that they are doing in implementing their own large language model because it’s a process, right There’s a lot of there’s more unknowns than knowns, and in partnering with our customers and many of our partners as well has been very rewarding as we all learn together vendor solutions, right? And that’s just increasing the trust in AI because as part of that process, we need to talk and document all kinds of potential usages of AI, the good ones and the bad ones. And obviously ethics is something that’s top of mind for everybody. As I mentioned earlier, we don’t see AI as a replacement. We see it as a tool that will help testers to keep up with the… fast paced code of delivery. Otherwise, if you have developers using AI to build code and you are still testing code without AI, it’s not sustainable. You’re not going to be able to keep up with that volume of new code of new application features coming down the pike, You’re going to slow down the whole pipeline that’s not, excuse me, that’s not something that anybody wants. We really see the best use of AI is really when testers are helping AI track along. So use cases where AI just do their thing on their own. Like I said, earlier, does not seem to work for testers. It really needs to be used in conjunction with the tester on a step… by step basis. Not just like human in the loop type of mentality, but it’s you know, tester taking AI by the hand and then taking advantage of the benefits and the speed that AI can do certain tasks based on what the tester is asking it to do. So that seems to be the, for now at least that based on the level of maturity of the large language model, seems to be the best, not just the best output but really the best situation where you have high quality output that are produced in bite size chunks that the tester can look at it, judge if it’s good or bad, and develop some trust that in the next interactions with AI, the tester will be more open to embracing the output because he… or she has been able to analyze the output, make a judgment, you know, whether it’s good or bad or could be better et cetera. And then move along. So that seems to be the really the sweet spot at least for now. And when we look forward in 20 24, we see that because of, those use cases, having the tester work more closely with AI, we see that, they will develop more comfort in working with generative AI technologies. We’re seeing that for the, in our own customer base that have been working with us for the past year as we were learning throughout all those experiments, If we look at them now, they’re much more comfortable. They’re much more reliant on Gen AI to help them on their day to day It’s just a matter of like I said, polishing the use cases so that the user experience in this case, the tester’s experience… is more seamless and more intuitive for them. And like I said, just now like invite give them output in bite sized chunks that they can digest and then develop that trust and confidence specialization in large language models is something that’s really starting up really fast. Many of the global systems integrators, our partners, they have developed in the process of implementing specialized large language models, for example for banking, financial services, for the airline industry, for e commerce, cruise lines, right? They have trained and fine tuned models that are specific to those industries. And now they’re using those models to augment their teams that are doing testing in those different industries. So, tremendous benefits that they’re seeing… already. And obviously, you know, when we connect our tools at Catalan to those specialized large language models, the quality of the output is just much more aligned with the industry itself, right? That the team is working on, be it banking financial services, and so on. And then that skepticism that exists, you know, today is really kind of fading away if you will. People are being more open to adopt AI powered capabilities as they start seeing results, practical results Again, going away from that realm of possibilities and showing success stories that help organizations relate. Oh, like they were successful at this use case. That’s very similar to what we’re facing here and so on. And so forth. So we see that all the experimental phases is sort of it’s never going to finish. It’s never going to go away. But that early experimentation phase where people were just trying to see, to have a feel for what AI can do is moving or transforming to more of a foundational building AI to the foundation of the product of the capability of the features so that the teams and the end user can gain the value. The additional value Be it productivity efficiency, et cetera, that everybody expects through AI. So really excited for the year that we’re in right now and looking forward to seeing what, you know, how fast things move in the next time we meet here. All right. That was it for me today. I hope this was valuable, you know, Lessons learned, sharing experience and let’s open it up for questions and see what are you guys seeing out there?

26:02 | Adam Creamer
All right. Great session by Alex there. Thank you, sir. So we have a few questions rolling in The first one here. Ai can be good at learning specific patterns. But what happens for something new or different situations? How does AI react to that?

26:22 | Alex Martins
Yeah, it depends on the kind of AI that we’re talking about, right? If we’re talking about specifically generative AI, then it needs to understand, It needs to have a certain data set, right? That it has been trained on. And based on that, it will start predicting the next things to be produced, the next data to be produced, be it text or video or image. So, it really in the generative AI realm, if you don’t have quality data, It’s a problem for generative AI, right? It kind of ties back to what I was saying, Many of our customers are deploying their own large language models and then in pilot projects to train them and fine tune them with their own institutional knowledge, right? And data and all that. And that’s great. I mean, it needs to happen… But people need to realize that it is quite an undertaking because doing all that fine tuning and training will require a lot of effort on their part, right? With SME knowledge on actual testing. And then The quality, the initial quality will likely not be great of the data generated from it. And that’s where the ongoing iterative fine tuning will have to happen. So, this illusion that, oh, we’re going to deploy a large language model and we expect extremely high quality Data being generated based on how our own business runs. I mean, that’s not how it should be, But I don’t think a lot of people are realizing that. So, yeah, it’s an interesting time.

28:12 | Adam Creamer
Yeah, it seems like things are always changing And I know we have about a minute left here, but you mentioned early on that developers in QA tend to approach AI a little bit differently. And Carlos had a question here that said like, what are your thoughts on developers leveraging AI to help with developing some of the testing cycles? Like unit tests, regression, tests, things like that?

28:35 | Alex Martins
Absolutely. That’s a great use case. And I think just to kind of be clear test automation in test automation, I think the use cases, the development use cases, they convert very well to test automation in general, right? Because it’s about generating code et cetera. So absolutely, if you have the ability to leverage an output, right? Let’s say a unit test that the Gen AI created for a developer, Taking that and expanding on it to build, you know, bigger testing scenarios that will be automated. It’s great. I think where we’re still lacking a bit is more is better ways of thinking about Gen AI use cases for manual testing for testing. In general, There’s a lot going on in terms of experimentation, but we’re still struggling a bit all of us as an industry to measure actual value.

29:34 | Adam Creamer
Yeah, absolutely. Well, we’re out of time and there are actually quite a few more questions that rolled in. So, Alex, if you hang out in the chat or head over to the Katalon booth, we’ll be happy to get those answered for you guys. But, Alex, thank you. Thank you very much.