COEs: Optimizing Infrastructure to Support Internal Customers

Video Transcript

0:00 | Matt Klassen
good morning. Good afternoon. And welcome to the next session for track one. Specifically, we are gonna get started here in just a moment. I’m gonna give a few people just a minute to join. But I have with me today, Ringo Perez from at T and I think it’s gonna be a fantastic session with Ringo and we’re gonna be talking about how he’s built a center of excellence as a part of a team, kind of his journey. And so again, my name is matt claussen. I run marketing here. I have a passion not just for software but for mobile as well. And have worked with a lot of companies in various aspects of their software development challenges journey. And I think again, one of those things that you see is how do you scale from the individual team? How do you create an economy of scale with a larger organization? And my past company? Parasoft, we saw this very often as it related to the test to creating virtual apis, the ability to create synthetic data, test data for applications, so that you could, you know, the development teams could shift left, have availability to do the testing of their applications even when those API services or data wasn’t available. And I think mobile and mobile devices and the mobile device lab is one of those things that’s very similar organizations that are trying to scale create an economy of scale… ultimately, a culture of excellence, a culture of accelerating app… dev and mobile development, find this really difficult because individual teams each setting up their own lab doesn’t make a whole lot of sense. But when you can create a center of excellence around QA or in this case, mobile QA. And at the center of that, have a mobile device lab, you can create this really cool environment for the organization and mobile app dev teams to sort of grow and thrive. So with that, that’s kind of my intro to the session that’s what we’re here to talk about today. So welcome, Ringo. Thank you for joining us today. We really appreciate it if you could just start with a kind of an introduction of yourself and your role at T. Sure.

2:41 | Ringo Perez
Thank you, matt. Well, hello, everyone. I am Ringo Perez, been working with at T since 1996 and started being involved with at T mobile center of excellence around 2012. When we were presented with the problem on how we can test mobile applications that we are developing during that time, you know, we were tasked, you know, how can we develop and test, you know, real devices, you know, that can be accessed remotely not just within here in the us but also, you know, around the world where a lot of our developers and testers are located?

3:31 | Matt Klassen
Yeah, that’s awesome. That’s awesome. So, I’m guessing a lot has changed since 2012. Maybe take us through a little bit of your journey kind of where, you know, kind of where you started some of the key challenges. And then, you know, maybe, how that journey has kind of gone to some of the highlights in terms of, you know, solving challenges and then more challenges and then kind of bringing us up to today. Yeah.

3:53 | Ringo Perez
So, so basically, as I’ve said, you know, the idea was presented to us, you know, during that time 2012, you know, when all this idea about mobile first, you know, we were asked, you know, as part of a team heading pioneering the mobile first idea, you know, focusing basically for our team on mobile testing aspect, you know, how are we going to solve that problem on providing capability for the different organizations? The different teams? You know, that will require real device testing? So we presented our idea, you know, that we were working on during that time and we were awarded funding, you know, to really work on that idea, hone that idea. We started with just, you know, setting up our own lab. I don’t know if you guys have heard about zaptest. We started with that, which was a good basic stuff that we were able, to use but it was not stable. It was not reliable. And, you know, as we plan to connect, you know, resources from around the world, India specifically, you know, to a lab here in the us. So we actually look on, several available vendors during that time that can help us, with our problem. You know, we tested and look on zaptest. I don’t know if you guys have heard of that monkeytalk, perfecto, mobile C test, eggplant, mobilelabs, and then we ended up with three tools actually because, you know, at that point in time, I believe, you know, these are the best tools that can solve some of the problems that we have perfecto mobile eggplant and mobilelabs. So those are the tools that we actually started with our mobile center of excellence.

5:57 | Matt Klassen
What, and what year was that, you selected those tools? Was that, so, that was about that?

6:05 | Ringo Perez
Was about 2012. So that’s 11 years ago, just 11 years.

6:10 | Matt Klassen
Ago. OK. So you actually started the journey with mobilelabs back then, which by the way, for those that don’t know, kobiton purchased mobilelabs several years ago. So have merged those product lines. But so then, OK, so then take us through you know, those early days. You now, have you’ve selected some tools you’re using them, you’re trying to scale them within your organization. You know, can you talk about what, you know a little bit a little bit about, that journey? And, and then, maybe, you know, where have you come right? Like, where are you today, with your current environment? Describe that a little bit?

6:47 | Ringo Perez
Yeah. So, so as I said, you know, we started with just the capability to have our developers and testers, you know, access, on real devices that they can use anytime anywhere. So those are an amazing capability during that time that’s thinking about just 11 years ago, not even thinking about automation at that time. No automation, just, you know, for us to be able to get a hold of devices, you know, from my desktop, right? So, that was the capability that was, the solution that we have, you know, solved during that time. So we, you know, we started, you know, able to start with our mobile automation when we were using actually, you know, as I’ve said, you know, some of the tools that we have selected, you know, perfecto mobile. They are advanced compared to everyone during that time, slowly mobile labs, you know, as part of the kobiton products now was able to catch up on some of the object based type of detection that we were able to leverage. We also have eggplant which is on our tool stock even right now for image based type of detection, which was unique offering during that time. So we eventually move out of perfecto mobile for it becomes too expensive for us to support. We were able to find a replacement, a solution of combo with katalon, and kobiton, you know, so we added kobiton, we have boba labs, but we also added kobiton during that time. So it was not a perfect solution. Then that was probably six years ago. It took us about two years to be able to figure it out and we are still discovering a lot of new things but we believe it was a right decision then and it is still the right decision for us right now. So we have convinced actually our community to fully utilize the combo and be able to create our automation framework using katalon and kobiton. You know, for us to integrate with our mobile testing infrastructure?

9:02 | Matt Klassen
So that’s interesting. So you use describe that a little bit more detail because I think this is one of the things that a lot of organizations struggle with is automation, right? So getting access to real devices in a timely manner for manual testing is one thing, right? But then being able to scale that. And do you know, automation and parallel at scale is also challenging, there’s a lot of different potential ways that could be done with appium and other frameworks. So describe a little bit more about how do you use katalon with kobiton together? That’s interesting. Yeah.

9:40 | Ringo Perez
So when we were able to stumble with this combo of katalon and kobiton, I believe, you know, there were a lot of integration happening during that time between the two. That’s why we selected them. Katalon was able to provide us the framework, our testing automation framework because kobiton was just concentrating on providing the mobile labs capability. They have some scripting capability as well automation capability. But basically the whole framework, you know, can be done on the katalon side which not, which also provide, you know, the capability to automate your web, your API and then mobile at the same time. So we know that, you know, we’re not just dealing with just mobile right now. We’re also dealing with all those different ways of automation talking about web and API. So a lot of the issues definitely that we encounter is not because the tool can do the job. It’s how we configure how, you know, the different networks are communicating. Most of the issues that we have are network blockage, right? You know, we have, you know, we’re talking about from our network to, you know, katalon or kobiton network, you know, how are you going to be able to integrate those with, you know, thinking about all the different securities as well that are being, you know, blocking, you know, those capabilities. So those are the challenges, you know, but basically, you know, those are the challenges that I think, you know, we need to be able to understand, we need to be able… to figure out so that, you know, we will be able to, you know, to do all those capabilities that we intend to do yep.

11:44 | Matt Klassen
Yep. No, that makes sense. I think that’s a good insight. I think the point here is that there, there are multiple ways that, you know, we’ve seen organizations have success in terms of scaling automation. And so if you are out there listening today and you haven’t looked at katalon, as a way to sort of help and, or integrate with kobiton, I think that’s you know, a good option to look at. In addition, of course, we do offer some other capabilities around, you know, generating appium scripts from manual sessions as well as you can. There’s other automation frameworks you can bring to play. But let’s talk a little bit about your team, supports your support for teams because this is, you know, this is one of the keys here. It’s not just setting up an environment but setting up an environment that scales across the needs of your organization. So how many can you talk a little bit about how many teams you support or how many dev teams or QA testers that you support around the world currently?

12:45 | Ringo Perez
Yeah. So within the at T enterprise right now, we’re actually supporting 125 business units. So that can equate to, you know, several applications that we’re supporting using, our platform. Yeah. So, so, you know, they definitely, they come and go, right? You know, they wanted to use this, you know, they will invest. So, those are the number of business units that we’re supporting right now? Yeah. So, so it’s, I have about five resources right now, you know, supporting the onboarding, supporting the different issues that we’re having between connectivities supporting any, you know, scripting, probably issues that they have configuration, those kind of things. We’re also developing… some capabilities on how we will support our, internal users within at T. So we are creating some tools as well. We’re using… some of the automation that we’ve learned from the tool that we have. Those are some things that we’re doing that’s the team composition that we have. We were able to scale down as we continue to understand what we really need. We were able to scale down as well. Some of the tools as I’ve said we cut down on the tools because now we understand more and more what we really need, you know, just the right tools tool stock that we need.

14:40 | Matt Klassen
Yeah, that’s very interesting. So I have a couple more follow on questions. So one is, you know, with that diverse set of business units and app apps, and in terms of, you know, needs from those groups, how many different device configurations do you have to support, right? How many different I think this, is a common question amongst many organizations trying to scale is like the difficulty, in supporting, you know, different types of devices and configurations. But do you guys have a magic number per app or per, you know, that the organization has kind of come up with? Yeah.

15:15 | Ringo Perez
So, so the composition of our, so, so thinking about application first, right? We have internal applications which is being utilized just internally within the company for the employees. And then we have external applications as well that are being used by our customers. So we need to think about that. We need to think about, you know, the devices that are only for internal use, which, you know, have different securities, right? We need to think about mdms, you know, those kind of things. And then we need to think about, you know, the external customer, what does that look like? So currently, we have about more than 200 at least devices that we have on our platform between kobiton and mobilelabs, that we have right now that are being utilized. So, yeah. So those are the different ranges and, you know, how are we going to, how are we going to configure them based on the application needs? Right? It’s not just, you know, it’s not just one kind of application but different types of customers as well. And then on top of that, there are some needs specifically to that devices, you know, depending on how the application is performing or how the application is being utilized. So there are different configurations, there are different ways of us, you know, scaling our devices… new iphones, new androids. So we need to think about, you know, what are we going to add? Which OS are we going to support? So when are we going to, when are we going to upgrade? What are the devices that we need to retire those kind of things? So yeah, there’s a lot of things that.

17:13 | Matt Klassen
yeah, yeah. I can imagine. I mean, we obviously deal with many of those same things because we have our own public cloud offering as well, right? So, I think that’s maybe important distinction here is that when one of the things that at T and some other enterprises want and, or need is the ability to manage their own device lab, right? And so kobiton doesn’t just have a public cloud environment like many or like a few others in the market, but we actually offer the ability for organizations like at T to scale create and scale their own device lab using either our carts or we have rack based sort of configuration as well. And then does at T connect, do you run standalone from a software perspective like the portal into the devices or are you running in our cloud just maybe talk a little bit about that? I think I’m pretty certain you’re running in our cloud, correct?

18:12 | Ringo Perez
Yeah, sure. Most of our, most of our devices are actually connected in your cloud. So we have two, we have the kobiton cloud and then we have the mobile labs cloud. You know, we still have both. And then we also have some of our teams are actually still using the cart, the mobile labs card. It’s still, you know, there are some teams that they need to have their own lab which, you know, they still have the cloud because of, you know, network needs and those kind of things and security, you know, just that application needs. So, we still have those. So, we have those options for them. So, yeah. So most but probably 90 percent of our, 90 percent of our devices are in kobiton cloud and mobile labs cloud.

19:09 | Matt Klassen
OK. Yeah. So I think that’s I think what you’re you know, one of the things that you’re touching on is that as you want to scale a center of excellence, right? Generally, one size doesn’t fit all meaning, you’re going to have a diverse set of needs and capabilities across those different apps. And again, you said there’s internal apps, there’s external apps, obviously internal apps, maybe you don’t need as many device configurations, external apps, consumer based obviously need probably more. One of the big considerations, you mentioned, is network… and then also specific security needs, right? So, so security is a big factor. So if we were to, I want to dig in on another thing, which is how do you, what are your measures of success? So as you start thinking about like since, you’re not an individual dev team, you represent and provide a service to a whole set of, you know, application teams, dev teams, you know, what?

20:10 | Matt Klassen
what are the, you know, KPIs? How do you manage the center of excellence? What are KPIs that you would use in terms of like key metrics?

20:19 | Ringo Perez
Yeah. So when we started, we just think about the growth of users adoption. You know, how are they going to adopt our platform? So we monitor that the users, we monitor the usage, we monitor, you know, how many are using the devices itself. And then, you know, we project that on a year by year basis, how many business units are coming in, you know, so, we work really hard during that time because, you know, we have there’s a lot of organizations, a lot of teams, you know, doesn’t know that we exist. So that’s where we started. And then, and then we started, you know, monitoring as well. You know, specifically, how are they utilizing it manually or automation? So we need to figure out, you know, are they just using it for manual usage or automation? So we’re checking those as part of our KPI. So there’s a lot of ways, you know, and then we try to mature by, you know, making sure that, you know, we will be able, we will be able to help them, you know, add this as part of their CI CI CD dev ops process to make sure that, you know, we are being incorporated in that process, you know, to help them out, with their automation. So, yeah. So, we started with just, you know, checking, the users usage. And then, you know, eventually, you know, we’re able to check, you know, how are they utilizing? You know, the platform?

22:07 | Matt Klassen
Yep. I think that’s yeah, that’s really key as well, which is, I think what you, what you’ve indicated and we find as well with many of, our customers, which is there’s this maturity sort of journey that you go on and the KPIs will change and morph and get more sophisticated. And, you know, from the basics, of utilization adoption to how organizations are utilizing. And then I think you mentioned another thing which is that transition to automation, right? And, and integration back into, right? The core CI CD pipeline and, or tool chain, which is key. So that’s awesome. So how, so, how are you guys funded? Like, how do you, as a center of excellence like, to, does each app team? Do they have to fund you or, you know, is there a chargeback for your services? I mean, how do you receive funding and how have you scaled that over the years?

23:08 | Ringo Perez
Yeah. So we function, as a service, within the company. So that’s one of the things that we try to eliminate from the different organizations that you know, they need to think about, you know, how are they going to be funded so that they’ll be able to utilize our platform? So, what we decided to do is let’s handle all of that so that we can eliminate that, you know, another hurdle from the different organization that will be utilizing it. Let’s you know, centralize everything, you know, we provide the licensing, we provide, the resources, you know, on how to maintain, manage it. Let’s just allow them, you know, on how they will be able to utilize, our platform. So that’s what we did. We’re you know, we come back and forth, you know, are we going to charge them because, you know, this is being charged on our organization? So I think there is that kind of things that are happening behind the scenes. But basically, you know, for us, I think we try to eliminate that, but definitely, you know, it depends on the company, the organization that can be done as well, that there will be a chargeback, that can be provided. So, you know, we’ve been thinking about those because how can we grow our platform? If we don’t have enough funding, right? We need to make sure that, you know, that the other organizations who are benefiting from our platform, that they will also include this on their budget so that, you know, we will be able to grow the platform?

24:49 | Matt Klassen
Yeah. So that’s.

24:50 | Ringo Perez
so, yeah.

24:53 | Matt Klassen
Well, no finish if you have more to say, yeah, finish on that. No, no.

24:55 | Ringo Perez
No, that’s basically so.

24:58 | Matt Klassen
Well, the follow on question would be OK. So what is an, you know, have you had to ever do an Roi, some sort of a return on investment, you know, calculation or, you know, justification or business document through the years to sort of, you know, continue to expand or, you know, I’m just curious if that’s been something you’ve done?

25:21 | Ringo Perez
Yeah, that’s challenging if, you know, we’re just providing the service. So, what we normally do is that we survey those important applications or users and we ask them, you know, what does this look like to you guys, when we provide this service to you? How can you help us monetize this? You know, how much is the company saving? You know, because you guys are utilizing this platform, so we have a way of monetizing it and understanding, you know, the return on investment because, what we’re providing them is actually a service that they are supposed, to pay on their own. But, you know, what we’re asking them in return is that, you know, provide us, you know, some of the data, some of this information so that we will be able to continue to justify, you know, what we’re doing. So, so we were able to, we were able to provide that to our leadership continuously every year. You know, that this is what we save because this particular application utilize our platform. This is how, you know, how much savings that are being realized and unrealized. You know, because they are preventing production issues, right? Because they utilize our platform. So we’re able to figure out those, you know, by, you know, just asking the users, the application users that’s.

26:56 | Matt Klassen
awesome. That’s awesome. OK. So I did want to open up for Q a. So I’m going to just verbally say I put it in the chat but, you know, enter your questions if you have any questions for Ringo. But actually since I’m not seeing any yet and I didn’t open this up earlier, I did in the chat but didn’t verbally ask you guys as an audience, what I’m going to do is I’m going to go ahead and do a poll question. So I’m going to open up this poll and it should come in your chat. So it should be in the chat at this time and… we can use let’s see. Oh, here we go. So go ahead, and, or there’s a polls area as well that you should be able to get into and start voting. So I’m sharing it. Now it should be live. So hopefully people can figure out how to vote. I’ll leave this open for it’s. Going to be open for another 10 seconds or so. And all right, I’m starting to see people vote. So the question is about where are you on your mobile app, QA, center of excellence journey… and… we are going to be wrapping this up. I do have one question. So right now, it looks like there’s you know, several people that don’t have plans, some that are trying to scale it, other organizations that have just started. So, I’m going to go to Q a. There is one more question. So last question and then we’re going to have to transition to the next session, which is any ideas or best practices you have used to help promote awareness of your center of excellence.

28:47 | Ringo Perez
So what we did especially at the start is that we actually went to some of the at T offices and set up a working session. So we presented the tools how to use it. And then, you know, before that, you know, we asked them, you know, to make sure that they will be able to have an access on kobiton or mobile labs, so that they will be able to practice it. And then we have some problems that, you know, they need to solve. So that’s what we did before. But right now, you know, we’re having sessions every month, you know, to provide new features. We’re asking them, you know, anything that, you know, that we can help with. So we have a monthly session to make sure that our users will be able to, you know, will be able to use our platform or if they have any problem, we invite some of our, you know, kobiton resources so that they’ll be able to answer, you know, some of the questions that they have. So we try to make sure that, you know, they are engaged. I think that’s the key is that make sure that, you know, our users will always be engaged.

30:15 | Matt Klassen
Yep. No, that’s awesome. That’s awesome. So, hey, thank you so much for your time today, Ringo. Thanks everybody who joined. And I think people are now transitioning to the next session. So if you haven’t you can transition to that, but I really do appreciate it, Ringo, and this of course will be recorded and available on demand as well. So some of you in fact might be listening on demand, but thanks very much appreciate it very good.

30:38 | Ringo Perez
All right. Thanks, matt. Thank you.