COEs: Optimizing Infrastructure to Support Internal Customers

Video Transcript

0:00:03.445,0:00:06.333

Good morning. Good afternoon, and welcome

0:00:06.333,0:00:12.733

to the next session for track one, specifically, We are gonna get started here in

0:00:12.733,0:00:17.493

just a moment. I’m gonna give a few people just a minute minute to join

0:00:17.493,0:00:19.913

but I have with me today

0:00:22.166,0:00:24.166

Ringo Perez from AT&T

0:00:24.166,0:00:30.134

 and I think it is going to be a fantastic session with Ringo

0:00:30.134,0:00:35.867

And we’re going to going to be talking about how he has built a center of excellence

0:00:35.867,0:00:39.367

as a part of a team, kind of his journey

0:00:39.915,0:00:42.929

And and so

0:00:42.929,0:00:47.439

Again, my name is Matt Klassen. I run marketing here, but have

0:00:47.439,0:00:51.807

passion, not just for software, but for mobile as well.

0:00:51.807,0:00:55.677

And have worked with a lot of companies lenders

0:00:55.677,0:00:58.417

years and various aspects of their software development,

0:00:58.417,0:01:01.497

challenges, journey, and I think it

0:01:01.497,0:01:05.417

I think again. One of those things that you see is how do you

0:01:05.417,0:01:07.437

scale from an individual team

0:01:07.437,0:01:11.567

how do you create an economy of scale with or larger organization?

0:01:13.821,0:01:17.206

In my pass parasoft, we saw this

0:01:17.206,0:01:20.776

very often as it related to the test

0:01:20.776,0:01:28.884

to creating virtual Apis, the ability to create some synthetic.

0:01:28.884,0:01:32.534

Data, test data for applications so that you could

0:01:32.534,0:01:35.794

you know, the development teams could shift left.

0:01:35.794,0:01:40.244

Have availability to do the testing of their applications, even

0:01:40.244,0:01:44.866

when those Api services or data wasn’t available.

0:01:44.866,0:01:46.186

And I think mobile

0:01:46.186,0:01:50.709

and mobile devices and the mobile device app is one of the locations it’s very similar

0:01:50.709,0:01:54.489

organizations that are trying to scale, create an economy of scale,

0:01:54.489,0:01:59.349

a ultimately a culture of pulp

0:02:00.324,0:02:06.624

of accelerating App dev and mobile development,

0:02:06.624,0:02:10.414

find this really difficult because individual teams

0:02:10.414,0:02:13.324

each setting up their own lab doesn’t make a whole lot of sense.

0:02:13.324,0:02:17.394

But when you can create a center of excellence around

0:02:17.394,0:02:20.744

Qa or in this case, mobile qa,

0:02:20.744,0:02:23.764

and at the center of that, have a mobile device lab.

0:02:23.764,0:02:28.734

You can create this really cool environment for the organization and mobile app

0:02:28.734,0:02:33.152

dev teams to sort of grow and thrive. So with that, that’s kind of my intro of the session.

0:02:33.152,0:02:37.317

That’s we’re here to talk about today. So welcome Ring. Thank you

0:02:37.653,0:02:40.673

for joining us today. Really really appreciate it. We’ll start with

0:02:40.673,0:02:47.353

kind of an introduction of yourself and role at At and T, Sure. Thank you, Matt. Well,

0:02:47.353,0:02:51.853

Hello, everyone. I am Ren perez

0:02:51.853,0:03:00.991

been working with At and T since ninety ninety six, and started being involved with At and T Mobiles center of Excellence.

0:03:00.991,0:03:05.421

Around two thousand twelve when we were presented with the problem

0:03:05.421,0:03:09.011

on how we can test mobile applications

0:03:09.011,0:03:10.961

that we are developing

0:03:10.961,0:03:15.191

during that time, you know, we were a task, you know,

0:03:16.972,0:03:21.121

how can we develop and testing on real devices.

0:03:21.121,0:03:23.781

You know, that can be access remote be.

0:03:24.000,0:03:26.955

Not just within hear in the Us, but also

0:03:26.955,0:03:31.145

you know around the world where you know, By developers and testers are

0:03:36.000,0:03:39.560

yeah. That’s awesome. That’s awesome. So I’m guessing a lot is changed

0:03:39.560,0:03:43.810

since two thousand twelve. Maybe take us through a little bit of your journey

0:03:43.968,0:03:44.948

kind of where kind

0:03:44.948,0:03:48.791

kinda where you started, some of the key challenges and then, you know, maybe how that

0:03:48.791,0:03:53.581

how that journey has kinda gone of some of the highlights in terms of, you know, solving challenges and then more challenge and

0:03:53.749,0:03:56.999

and then kinda bringing us up to today.

0:03:57.369,0:04:02.339

Yeah. So so, basically, as I’ve said, you know, the idea was presented to us

0:04:02.339,0:04:07.309

you know, during that time, twenty twelve, know when all this idea about

0:04:07.309,0:04:11.619

mobile mobile first. You know, we were asked

0:04:11.619,0:04:14.979

you know, a part of a team

0:04:14.979,0:04:19.309

head in pioneering the mobile first idea

0:04:19.309,0:04:24.279

you know, focusing basically for our team on mobile testing as

0:04:25.283,0:04:29.593

you know, how are we going to solve that problem on

0:04:29.593,0:04:34.503

providing capability for the different organizations, the

0:04:34.503,0:04:38.603

different teams, you know, that will require real device testing.

0:04:38.603,0:04:43.063

So we presented our idea

0:04:43.063,0:04:43.963

you know, that you

0:04:43.963,0:04:48.153

that we were working on during that time, and we were awarded

0:04:48.153,0:04:52.193

funding, you know to really work on that idea on that idea,

0:04:52.193,0:04:55.213

we started… We just, you know, setting up

0:04:55.213,0:04:59.973

our own lab… I don’t know if you guys have heard about Zap test.

0:05:00.000,0:05:03.260

We started with that, which was a good basic

0:05:03.260,0:05:07.380

stuff that we were able to to use, but it was not

0:05:07.380,0:05:11.760

stable. It was not reliable. And, you know, as we

0:05:11.760,0:05:14.780

plant to connect resources,

0:05:14.780,0:05:19.240

from around the world India, specifically, you know to a ;ab here in the Us.

0:05:19.482,0:05:24.092

So we actually do on several several available

0:05:24.260,0:05:29.230

vendors during that time that can help us with our with our problem.

0:05:29.230,0:05:34.200

You we tested and look on Sop. test I don’t know if you guys have heard

0:05:34.436,0:05:38.446

at monkey park, perfect well, test,

0:05:38.446,0:05:41.096

eggplant, mobile labs,

0:05:41.096,0:05:44.146

and then we ended up with three tools actually because

0:05:44.146,0:05:48.747

you know, at that point in time, I we believe, you know, these are the best tools that can

0:05:48.747,0:05:53.357

solve some of the problems that you have perfect mobile, eggplant and

0:05:53.357,0:05:54.727

and mobile loves

0:05:54.727,0:05:58.437

those are the tools that we actually started with.

0:05:58.437,0:06:00.757

With our Bog center of excellence.

0:06:00.757,0:06:04.355

What and what year was that that you

0:06:04.355,0:06:06.945

you’ve selected those tools. Was that

0:06:06.945,0:06:08.805

that that was another owner beyond.

0:06:08.805,0:06:10.925

That was about two thousand

0:06:11.297,0:06:15.037

twelve. So there’s some eleven years ago. Yeah. Just eleven years ago. Okay.

0:06:15.037,0:06:18.927

You actually started the journey with mobile labs back then, which by the way, for those that don’t know.

0:06:18.927,0:06:20.737

Kobiton purchased

0:06:20.737,0:06:23.757

mobile labs several years ago. So

0:06:23.757,0:06:28.727

have merged those product lines. But… So then… Okay. So then take us through

0:06:28.727,0:06:32.377

you know those early days, you now have… You’ve selected

0:06:32.377,0:06:36.407

some tools, you’re using them, You’re trying to scale them within your organization.

0:06:36.913,0:06:41.883

You know, can you talk about what, you know, a little bit a little bit about that that journey and

0:06:41.929,0:06:44.089

And then maybe, you know,

0:06:44.782,0:06:49.368

where where have you come? Right? Like, where where are you today with with your current environment?

0:06:49.368,0:06:50.708

Describe that a little bit.

0:06:50.947,0:06:55.707

Yeah. So so as Said, you know, we started with just the capability to have our developers

0:06:55.927,0:06:59.037

and testers, access on

0:06:59.037,0:07:02.847

on real devices that they can use any time

0:07:02.997,0:07:07.967

anywhere. So those are an amazing capability during that time. That’s

0:07:08.315,0:07:11.515

thinking about just eleven years ago. Not even

0:07:11.515,0:07:16.485

thinking about automation, at that time, no auto automation, just, you know, for us to be able to

0:07:16.485,0:07:20.105

get a hold of devices, you know, from my desktop. Get

0:07:20.105,0:07:23.425

Right? So that that was the capability that was the

0:07:23.425,0:07:27.395

the solution that we have, you, solved

0:07:27.395,0:07:33.645

during that time, So we, you know, we started, you know, able to start with

0:07:34.873,0:07:36.773

our mobile automation,

0:07:37.042,0:07:39.802

when we were using App,

0:07:40.243,0:07:44.463

you know, as I’ve said, you know, some of the tools that we have selected. You know, perfecto mobile.

0:07:44.665,0:07:48.645

They’re are advance compared to everyone during that time.

0:07:48.645,0:07:53.465

Slowly mobile labs, you know, part of the Kobiton product

0:07:53.465,0:07:55.215

now was able to catch up.

0:07:55.354,0:07:56.854

On some of the object

0:07:56.854,0:08:00.924

base type of detection that we were able to leverage.

0:08:00.924,0:08:04.424

We also have Eggplant, which is on our tools

0:08:04.648,0:08:08.028

stock even right now. For image

0:08:08.396,0:08:12.686

base type of detection, which was unique offering during that time. So

0:08:12.740,0:08:15.010

We eventually move out

0:08:15.010,0:08:17.620

perfect before it becomes too expensive,

0:08:17.620,0:08:20.670

For us to support, we were able to find a replacement.

0:08:20.670,0:08:30.000

A solution of combo with Katalon and Kobiton. So we had Kobiton. We have mobile labs, but we also added Kobiton during that

0:08:30.000,0:08:37.900

time. So it was not a perfect solution, then that was probably six years ago, it took us about two years to be able to figure it out.

0:08:37.900,0:08:41.700

And we are still discovering a lot of new things.

0:08:41.700,0:08:46.550

But we believe it was a a right decision then, and it is still a right decision

0:08:46.550,0:08:48.700

for us right now.

0:08:48.700,0:08:51.720

So we have convinced after the our community

0:08:51.908,0:08:56.878

to fully utilize the combo and be able to create our automation framework

0:08:56.878,0:08:59.858

using Katalon and Kobiton.

0:09:00.000,0:09:01.530

You know, for us to integrate

0:09:01.530,0:09:04.710

with our mobile testing infrastructure.

0:09:04.710,0:09:09.680

So that’s interesting. So you use… Describe that a little bit more detail, because I think

0:09:09.680,0:09:13.390

this is… It’s one of the things that a lot of organizations struggle with is

0:09:13.390,0:09:16.600

is automation. Right? So getting access

0:09:16.600,0:09:21.420

to real devices in a timely manner for manual testing is one thing. Right?

0:09:21.911,0:09:24.361

But then being able to scale that

0:09:24.361,0:09:27.681

and do, you know, automation and parallel,

0:09:27.681,0:09:30.891

at scale is also challenging. There’s a lot of different

0:09:30.891,0:09:35.531

different potential ways that could be done with Appium and other frameworks

0:09:35.732,0:09:37.552

So describe a little bit

0:09:37.552,0:09:42.522

more about how how do you use Katalon with Kobiton together that’s

0:09:42.522,0:09:48.022

interesting. Yes. So when when

0:09:48.837,0:09:51.516

when we were able to stumble with

0:09:51.516,0:09:54.806

this combo Katalon and Kobiton. And I believe you know there were

0:09:54.806,0:09:57.856

a lot of integration happening during the time.

0:09:57.856,0:10:01.466

Between the two. That’s why we selected them.

0:10:01.466,0:10:05.340

Katalon was able to provide us the framework, a testing automation framework because

0:10:05.340,0:10:07.688

Kobiton was just concentrating on providing the

0:10:07.688,0:10:10.738

mobile labs capability. They have some

0:10:10.738,0:10:14.058

scripting, capability, as well

0:10:14.058,0:10:18.848

automation capability. But basically, the whole framework you know can be done on the Katalon

0:10:18.848,0:10:22.979

side, which not… Which also provides, you know, the capability to

0:10:22.979,0:10:28.189

automate web, your Api and then mobile at the same time.

0:10:28.189,0:10:31.719

So we know that, you know, we’re not just dealing with

0:10:31.719,0:10:33.030

just mobile right now.

0:10:33.030,0:10:36.380

Also dealing with all those different

0:10:36.380,0:10:43.960

ways of automation talking about web and Api. So

0:10:45.560,0:10:50.230

A lot of the a lot of the issues definitely that we encounter is

0:10:50.842,0:10:53.152

it’s not because the tool can

0:10:53.442,0:10:56.462

do the the job. It’s it’s

0:10:56.462,0:11:00.609

how we configure, how you, you know, the the different networks,

0:11:00.609,0:11:04.679

are communicating. Most of the agents that we have are network,

0:11:04.679,0:11:08.059

blockage. Right? You know, we have, you know, we’re talking about

0:11:08.496,0:11:10.826

from our network,

0:11:10.826,0:11:15.556

to, you know, to Katalon or Kobiton network, you, How are you gonna be able to integrate those

0:11:15.556,0:11:20.101

with, you know, thinking about all these different security

0:11:20.101,0:11:22.171

as well. That are being, you know, up as

0:11:22.171,0:11:26.321

blocking, you know, those capabilities. So

0:11:26.321,0:11:31.291

Those are the challenges, you know, But, basically, you know, those are the challenges that think

0:11:31.291,0:11:35.721

you know, we need to be able to understand. We need to be able to

0:11:35.721,0:11:37.131

to figure out

0:11:37.369,0:11:40.409

so that, you know, we will be able to…

0:11:41.283,0:11:43.263

You know, to do all those

0:11:43.633,0:11:46.133

capabilities that we intend to do.

0:11:47.847,0:11:51.354

Yep. Yep. No. That makes sense.

0:11:51.354,0:11:56.034

I think that’s a good insight. I think I think I think the point here is that there there are multiple ways.

0:11:56.034,0:12:00.594

That you know… We’ve seen organizations have success in terms of scaling

0:12:00.594,0:12:04.522

automation. And so if you are out there listening today, and you have

0:12:04.522,0:12:07.932

that Katalon  as as a as a way to sort of the

0:12:07.932,0:12:13.452

help and or integrate with Kobiton, think that’s, you know,

0:12:13.452,0:12:14.802

a good option

0:12:14.802,0:12:17.643

to look at in addition, of course,

0:12:17.643,0:12:21.603

we do offer some other capabilities around, you know, generating

0:12:21.872,0:12:25.322

Appium scripts from manual sessions,

0:12:25.490,0:12:29.500

as well as well as you can… There’s other automation frameworks you can bring to play. But

0:12:29.500,0:12:32.850

Let’s talk a little bit about your team supports, your support for teams because this is

0:12:33.085,0:12:38.005

this is one of the keys here. It’s not just setting up an environment, but setting up an environment that scales

0:12:38.005,0:12:41.115

across the needs of your organization. So how many

0:12:41.115,0:12:44.769

can you talk a little bit about how many teams you support or how many

0:12:44.769,0:12:48.699

dev teams or QA testers that you support around the world?

0:12:48.699,0:12:53.399

Currently? Yeah. So within the At and t enterprise

0:12:53.399,0:12:56.419

right now. Where support

0:12:56.419,0:13:00.759

hundred twenty five business units,

0:13:00.759,0:13:04.549

So that can equate to

0:13:04.549,0:13:08.979

you know, several applications that were supporting

0:13:08.979,0:13:16.769

using our our platform, Yeah. So so, you know, they

0:13:16.769,0:13:20.959

they definitely they come and go. Right? You know, They wanted to use this,

0:13:20.959,0:13:28.940

they will invest, those are the number of business units we are supporting right now

0:13:32.486,0:13:39.951

I have about five resources right now

0:13:39.951,0:13:49.066

You know, supporting boarding supporting the different issues that we’re having between connectivity.

0:13:49.066,0:13:53.401

Supporting any, you know, scripting, probably issues that they have

0:13:53.401,0:13:55.791

configuration on those kind of things

0:13:55.791,0:14:02.651

we’re also developing some some capabilities on how we will support our internal

0:14:02.651,0:14:09.632

users internal users within AT&T, umm so we are creating some tools as well. we are using

0:14:10.124,0:14:14.974

some of the automation that we learned from the tool that we have. So

0:14:15.512,0:14:19.278

Those are some things that we’re doing. That’s the team composition that

0:14:19.983,0:14:22.233

have. You know, we were able to scale

0:14:22.233,0:14:27.693

scale down, you know, as we continue to understand, you know, what we really need

0:14:28.048,0:14:31.248

We were able to scale down as well some of the tools some

At and T

0:14:31.248,0:14:33.580

said, you know, we cut down on the on the toes because now we

0:14:33.580,0:14:36.450

stand more and more you know what we really need, what

0:14:36.450,0:14:39.470

is, you know, just just just the right

0:14:39.470,0:14:42.400

at those stock that we need.

0:14:42.400,0:14:46.218

Yeah. Yeah. That’s very interesting. So I have a couple more follow on questions. So one is

0:14:46.218,0:14:50.138

you know, with that diverse set of business units in app,

0:14:50.273,0:14:54.463

apps and in terms of, you know, needs from those groups,

0:14:54.463,0:14:59.413

How many different device configurations do you have to support. Right? You have any different

0:15:00.993,0:15:05.533

think it’s a is a common question amongst many organizations try to scale is, like,

0:15:05.533,0:15:11.053

the difficulty in in supporting, you know, different types of devices

0:15:11.053,0:15:14.733

configurations. But do you guys have a magic number per app or per

0:15:14.733,0:15:18.473

know, that that the organization is has has kinda come up with.

0:15:19.413,0:15:23.723

Yeah. So so the composition of our… So so

0:15:23.723,0:15:26.743

thinking about application first. Right? We have

0:15:26.743,0:15:31.375

internal applications, which is being utilized just internally within the company.

0:15:31.375,0:15:34.965

For the employees. And then we have external

0:15:34.965,0:15:38.195

applications as well that are being used by our customers.

0:15:38.195,0:15:42.325

So we need to think about that. We need to think about, you know, the device

0:15:42.325,0:15:46.065

cases that are only for internal use, which

0:15:46.065,0:15:48.815

you know, have different securities. Right?

0:15:48.815,0:15:52.735

We need to think about Md m’s, you know, those kind things.

0:15:52.735,0:15:56.575

And then we need to think about, you know, the external customer, What does that look like?

0:15:56.710,0:16:00.500

So apparently, we have

0:16:00.500,0:16:05.450

about more than two hundred, at least devices that we have on our

0:16:05.450,0:16:08.950

on our call… Between Kobiton and mobile labs that we have right now,

0:16:09.151,0:16:13.971

that are being utilized. So… Yeah. So those are the different ranges.

0:16:13.971,0:16:18.521

And, you know, how are we going to… How are we going to configure them?

0:16:18.521,0:16:23.311

Based on the application needs. Right? It’s it’s it’s not just, you know,

0:16:23.311,0:16:24.631

It’s not just

0:16:24.631,0:16:28.281

one kind of application, but a different type

0:16:28.281,0:16:30.766

sub customers as well. And then and then on top of that,

0:16:30.766,0:16:34.096

there are some need specifically to that devices.

0:16:34.096,0:16:38.436

You know, depending on the how the application is performing.

0:16:38.436,0:16:41.066

Or the the how the application is

0:16:41.066,0:16:43.511

is being utilized. So

0:16:43.847,0:16:47.397

there are different configurations. There are different ways of

0:16:47.397,0:16:51.447

hope us, you know, scaling

0:16:51.649,0:16:58.399

our our devices, new iphones, new android,

0:16:58.399,0:17:02.439

So we need to to think about, you know, what are we going to add? Which

0:17:02.439,0:17:05.669

which os are we going to support.

0:17:05.669,0:17:10.169

So when are we going to when are we going to upgrade

0:17:12.000,0:17:15.440

what are the devices that we need to retire? Those kind of things. So

0:17:15.583,0:17:17.813

Yeah. There’s a lot of things that… Yeah.

0:17:18.000,0:17:21.680

Yeah. I can imagine. I mean, we we’re obviously dealing for many of the same

0:17:21.680,0:17:23.230

things is we have our own public

0:17:23.230,0:17:28.046

cloud offering as well. Right? So I think that’s maybe important to

0:17:28.046,0:17:34.496

distinction here is that when one of the things that At and T and some other enterprises

0:17:35.572,0:17:38.842

want and or need

0:17:38.842,0:17:43.362

is the ability to manage their own device lab. Right? And so Ko

0:17:43.362,0:17:45.682

doesn’t just have a public cloud environment

0:17:45.682,0:17:48.762

like like many or like, a few others in the market

0:17:48.762,0:17:53.732

But we actually offer the ability for organizations like At and T to

0:17:53.732,0:17:55.872

scale, create and scale their own.

0:17:55.872,0:17:59.982

Device lab. Using either our carts,

0:18:00.000,0:18:03.200

or we have rack based

0:18:03.350,0:18:05.390

sort of configuration as well.

0:18:05.390,0:18:10.360

And then does At and T connect… Do you run standalone from a software

0:18:10.360,0:18:15.060

perspective, like, the portal into the devices? Or are you running in our cloud? Just maybe talk a little bit about that? I think

0:18:15.060,0:18:16.920

pretty sure you’re running in our cloud. Correct? I’m

0:18:16.920,0:18:20.960

Yeah. Sure. Most of our most of our

0:18:21.162,0:18:25.682

devices are connected in your cloud. So we have two we have the Kobiton cloud and then

0:18:25.682,0:18:28.792

have the mobile Apps cloud.

0:18:28.792,0:18:32.642

You know, we still have both. And then we also have some

0:18:32.642,0:18:36.000

Some of our teams are actually still using the card.

0:18:36.000,0:18:40.800

The mobile labs card. It’s still… They… You know, there are some things that

0:18:40.800,0:18:43.370

they need to have their own lab

0:18:43.370,0:18:48.330

which, you know, they still have the the the cloud.

0:18:48.500,0:18:52.940

Because of you know, network, needs, and those can

0:18:53.142,0:18:55.042

things and security, you know, just

0:18:55.042,0:18:59.952

that that application needs. So we we still have those. So we have

0:19:00.000,0:19:03.292

we have those options for them.

0:19:03.292,0:19:07.532

So… Yeah. So most, but probably ninety percent of our

0:19:07.532,0:19:10.622

ninety percent of our devices are

0:19:10.622,0:19:13.972

in Kobiton cloud and and mobile apps cloud.

0:19:13.972,0:19:17.667

Okay. Okay. Yeah. So I think that’s

0:19:17.667,0:19:19.167

I think what what you’re

0:19:19.167,0:19:23.917

you know, one of the things that you’re touching on is that as you wanna scale,

0:19:23.917,0:19:25.367

a center of excellence. Right?

0:19:25.367,0:19:30.000

Generally, one size doesn’t fit all, meaning you’re gonna have a diverse set of needs and

0:19:30.000,0:19:35.560

abilities, across those different apps. In the… And again, you said there’s internal apps or external apps,

0:19:35.725,0:19:40.695

Obviously, internal apps, maybe you don’t need as many device configurations, external apps, consumer based

0:19:40.695,0:19:43.415

obviously, need probably more.

0:19:43.415,0:19:46.785

Are the big considerations you you mentioned is is network?

0:19:46.785,0:19:50.735

And and then also specific security needs. Right? So

0:19:50.735,0:19:56.045

So security is a big factor. So if we were to

0:19:56.045,0:19:58.755

I wanna dig in on another thing, which is

0:19:58.755,0:20:03.185

which is how do you what are your measures of success? So as you start thinking about, like,

0:20:03.185,0:20:08.155

you’re not a you’re not an individual dev team, you present and provide a service to a whole set.

0:20:08.155,0:20:12.835

Of, you know, application teams, dev teams,

0:20:12.835,0:20:14.805

You know, what are what are the

0:20:14.805,0:20:19.775

you know, Kpis, how do you manage the center of accents and what are Kpis that

0:20:19.775,0:20:20.275

would use in terms of, like,

0:20:20.275,0:20:23.685

P metrics. Yeah. So

0:20:23.685,0:20:28.385

when we started, we just think about the growth users

0:20:28.385,0:20:31.405

adoption, you know,

0:20:31.405,0:20:35.535

are they going to adopt our platform? So we monitor that?

0:20:35.535,0:20:39.335

The users. We we monitor the usage,

0:20:39.335,0:20:44.305

we monitor, you know, how many how many are using the devices

0:20:44.305,0:20:48.135

itself, and then, you know, we project that

0:20:48.135,0:20:52.775

on a year by year basis. How many business units are coming in?

0:20:52.775,0:20:53.660

You know. So we

0:20:53.660,0:21:00.764

we work really hard during that time because, you know, we have

0:21:00.764,0:21:05.734

There’s a lot of organizations, a lot of teams, you know doesn’t know that we exist.

0:21:05.734,0:21:08.754

So that’s where we started.

0:21:08.754,0:21:11.984

And then and then we

0:21:11.984,0:21:15.754

started, you know, monitoring as well, you know, specifically how

0:21:15.754,0:21:19.404

How are they utilizing it? Manually or automation,

0:21:19.404,0:21:23.474

So we need to figure out, you know, are they just using it for manual?

0:21:23.474,0:21:27.394

Usage or automation. So where we’re

0:21:27.394,0:21:31.434

we’re checking those as part of our Kpi

0:21:32.275,0:21:35.725

So there’s a lot of ways, you know, And then we

0:21:36.162,0:21:40.922

try to mature by, you know, making sure that, you know, we were be able

0:21:41.057,0:21:47.638

we will be able to help them you know, add this as part of their C Ci the devops ops process.

0:21:47.638,0:21:52.608

To make sure that, you know, we are being incorporated in that process, you know, to help them out.

0:21:52.608,0:21:55.948

With with automation. So

0:21:55.948,0:22:00.000

Yeah. So we… We we started with just, you know, checking the

0:22:00.000,0:22:04.730

the users, usage, and then, you know, eventually you know which

0:22:04.730,0:22:09.040

we’re we’re able to check, you know how how are they utilizing, you know,

0:22:09.040,0:22:13.180

the platform. Yep. Yep. I think that’s

0:22:13.180,0:22:17.090

Yeah. That’s really key as well, which is… I think what you what you’ve

0:22:17.228,0:22:20.968

indicated that we find as well with many of our our customers

0:22:20.968,0:22:23.268

which is there’s this maturity

0:22:23.536,0:22:26.396

sort of journey that you go on,

0:22:26.396,0:22:30.886

and the Kpis will change and more and get more sophisticated in

0:22:30.886,0:22:32.713

you know, from, you

0:22:32.713,0:22:34.764

and the basics of of of utilization adoption,

0:22:34.764,0:22:37.194

to how organizations are utilizing

0:22:37.194,0:22:42.164

And then I think you mentioned another thing, which is that transition to automation. Right?

0:22:42.366,0:22:46.266

And an integration back into

0:22:46.266,0:22:50.786

right? The core Ci pipeline and or tool chain, which is which is key.

0:22:50.786,0:22:54.646

So that’s that’s awesome. So how so how are you guys funded it? Like, how

0:22:54.646,0:22:58.896

how do you… As a center of excellence like, did to does each

0:22:58.896,0:23:02.926

app app team. Do they

0:23:02.926,0:23:05.946

you have to fund you or, you know, is there a charge back?

0:23:06.000,0:23:09.560

For your services? I mean, how how do you receive funding and how have you

0:23:09.560,0:23:11.250

scale that over over the years

0:23:11.250,0:23:15.560

Yeah. So we function as as as a service.

0:23:15.560,0:23:18.760

Within the company. So

0:23:18.760,0:23:22.890

That’s one of the things that we try to eliminate from the different inorganic

0:23:22.890,0:23:25.910

is that, you know, they need to think about, you know, how

0:23:25.910,0:23:30.400

how they going to be funded or they’ll be able to utilize our platform.

0:23:30.400,0:23:33.870

So what what we decided to boost let’s handle

0:23:33.870,0:23:38.360

all of that. So that we can eliminate that, you know, another hurdle though from

0:23:38.360,0:23:41.410

the different organization that could be utilizing it. Let’s

0:23:41.410,0:23:44.850

you know, centralized everything. You know, we provide

0:23:44.850,0:23:48.060

the licensing we provide the

0:23:48.269,0:23:51.359

the resources, you know, on how to maintain manage it,

0:23:51.359,0:23:57.570

Let’s just allow them, you know, on how they will be able to utilize our our platform,

0:23:57.570,0:24:00.360

form. So that’s what we did. We’re

0:24:00.360,0:24:03.183

you know, we come back and forth? And are we going to charge them? Because, you know, this

0:24:03.351,0:24:06.941

being charged on our organization.

0:24:08.756,0:24:12.016

So I think there is that kind of things that are happening

0:24:12.016,0:24:15.246

behind the scene, but basically, you know, for us,

0:24:15.246,0:24:18.656

I think we try to eliminate that. But

0:24:18.656,0:24:20.916

definitely, you know, it depends on…

0:24:20.916,0:24:23.936

The company the organization that can be done as well.

0:24:24.000,0:24:26.520

That there will be a charge back that can

0:24:26.520,0:24:29.610

that can be provided. So we

0:24:29.610,0:24:31.760

know, we’ve we’ve been we’ve been thinking about

0:24:31.760,0:24:35.110

others those, because how can we grow

0:24:35.110,0:24:38.380

our platform if we don’t have enough funding.

0:24:38.380,0:24:38.890

Right.

0:24:38.890,0:24:43.800

We need to make sure that, you know, that the other organizations who are benefit

0:24:43.938,0:24:48.783

from our platform that they will also include this under budget

0:24:48.783,0:24:51.133

so that, you know, we will be able to grow.

0:24:51.133,0:24:55.703

The platform. Yep. Yep. So that’s show Yep.

0:24:56.908,0:25:00.372

Well, no. Finish. If you more to say… Yeah. Finish on that. No.

0:25:00.372,0:25:04.772

That’s basically… So my… Well, the follow on question would be okay. So

0:25:04.772,0:25:07.972

what isn’t… You know, have you had to ever do an Roi?

0:25:08.107,0:25:11.157

Some sort of a return on investment.

0:25:11.157,0:25:16.127

You know, calculation or, you know, justification or business document.

0:25:16.688,0:25:18.328

Through the years

0:25:18.328,0:25:22.548

to sort of, you know, continue to expand or, you know, I’m just I’m just curious.

0:25:22.548,0:25:24.998

That’s if that’s been something you’ve done.

0:25:24.998,0:25:28.018

Yeah. That’s challenging. If

0:25:28.018,0:25:31.938

you know, we’re just providing the service. So what we normally do,

0:25:31.938,0:25:35.668

is that we survey

0:25:36.000,0:25:40.010

those important applications or users.

0:25:40.010,0:25:44.980

And we asked them, you know, what does this look like to you guys when we provide this

0:25:44.980,0:25:47.820

service to you. How

0:25:47.820,0:25:52.790

Can you help us monetize this? You know, how much is the company saving?

0:25:52.790,0:25:55.900

You know, because you guys are utilizing this platform.

0:25:55.900,0:25:58.510

So we have a way of

0:25:58.510,0:25:59.010

monetizing it and understanding, you know,

0:25:59.010,0:26:01.690

you know, the the return on investment,

0:26:01.851,0:26:06.000

because what prop… What we’re providing them is actually a a service

0:26:06.000,0:26:12.180

that they are supposed to to pay on their own. But, you know, what we’re

0:26:12.180,0:26:20.969

asking them in return is to provided us with some of the data and information

0:26:20.969,0:26:24.649

So that we will continue to be able to justify what we’re doing

0:26:24.649,0:26:30.000

So uh we were able to provide that to our leadership

0:26:30.000,0:26:34.850

Continuously every year. You know, that this is what we save because this

0:26:34.850,0:26:37.900

particular application utilize our platform.

0:26:37.900,0:26:40.450

This is how, you know,

0:26:40.450,0:26:45.410

how much savings that are being realized and not realized, you know, because they are

0:26:45.410,0:26:48.200

preventing production issues.

0:26:48.200,0:26:52.360

Right? Because they utilize our platform. So we’re able to figure out

0:26:52.360,0:26:56.160

those, you know, by, you know, just asking the

0:26:56.160,0:26:58.560

the users, the application users.

0:26:58.560,0:27:01.560

That’s awesome. That’s awesome.

0:27:01.560,0:27:05.870

Okay. So it I did wanna open it for for q and a, So I’m gonna just…

0:27:06.000,0:27:10.970

Verbally say I put it in the chat, but, you know, answer your questions, if you have any questions,

0:27:10.970,0:27:13.270

for Ringo, but actually, since

0:27:13.270,0:27:16.890

since I’m not seeing it yet, and I didn’t open this up earlier, I did in the chat, but

0:27:16.890,0:27:19.340

didn’t didn’t verbally ask you guys.

0:27:19.340,0:27:24.310

As an audience. What I’m gonna do is I’m gonna go ahead and do it a poll question.

0:27:25.217,0:27:26.477

So I’m gonna open up the…

0:27:26.477,0:27:30.597

This pole and it should come in your chat. So

0:27:30.597,0:27:36.837

it should be in the chat at this time,

0:27:36.837,0:27:43.587

and we can use… Let’s see. Oh, here we go.

0:27:43.587,0:27:46.077

So go ahead and and

0:27:46.077,0:27:48.147

or there’s a poles area or

0:27:48.147,0:27:51.207

as well that you should be able to get into

0:27:52.082,0:27:57.812

and start voting. So I’m sharing it now. It should be live. So

0:27:57.812,0:28:02.422

Hopefully, people can figure out how to vote. I’ll I’ll leave its open

0:28:02.422,0:28:06.932

for it’s gonna be open for another ten seconds or so.

0:28:07.704,0:28:08.744

And

0:28:08.744,0:28:11.884

alright. I’m starting to see people vote. So the question is about… Alright

0:28:11.884,0:28:15.284

Where are you on your mobile app Qa center of excellence journey?

0:28:15.880,0:28:17.340

And

0:28:17.340,0:28:20.600

We are gonna

0:28:20.600,0:28:25.300

be wrapping this up. I do have one question. So right now, it looks like there’s

0:28:25.300,0:28:28.670

you know, several people that don’t have plans, some that are trying to scale it.

0:28:28.670,0:28:33.550

Other organizations that have just started. So I’m gonna go to Q and A. There is one more

0:28:33.550,0:28:41.390

question So last question then we’re gonna have to transition the next session, which is

0:28:41.390,0:28:44.410

Any ideas or best practices, you have

0:28:44.410,0:28:47.580

use to help promote awareness of your center excellence.

0:28:47.580,0:28:51.560

So what we did especially at the start,

0:28:51.560,0:28:53.340

is that we

0:28:53.340,0:28:57.860

after actually went to some of the At t offices and and set

0:28:59.810,0:29:03.760

a working session, So we

0:29:03.760,0:29:07.290

we presented the tools, how to use it,

0:29:07.290,0:29:12.260

And then, you know, before that, you know, we ask them, you know, to make sure that they will be able to

0:29:12.260,0:29:16.900

have an access on on Kobiton or mobile labs.

0:29:16.900,0:29:20.760

So that they will be able to practice it. And then we have

0:29:20.760,0:29:23.780

some some some

0:29:23.780,0:29:28.420

some problems that, you know, they need to solve. So that’s what we did before, but

0:29:28.420,0:29:30.870

right now, you know, we’re having

0:29:30.870,0:29:32.620

sessions every month.

0:29:32.620,0:29:38.120

You know, to to provide new features,

0:29:38.120,0:29:41.950

we’re asking them, you know, anything that, you know,

0:29:41.950,0:29:43.530

that we can help with

0:29:43.530,0:29:48.020

So we have a monthly session to make sure that

0:29:48.020,0:29:52.240

our users will be able to, you know, I will be able to

0:29:52.240,0:29:56.220

to use our platform. Or if they have any

0:29:56.220,0:29:59.520

any problem. We invite someone our

0:29:59.520,0:30:02.540

you know, of resources.

0:30:02.540,0:30:06.850

So they’ll be able to answer, you know, some of the questions that they have. So we

0:30:06.850,0:30:08.920

we try to

0:30:10.325,0:30:14.425

to make sure that, you know, they are engaged. I think that’s the sucky key. Is that? Make sure

0:30:14.425,0:30:18.005

that you know, our users will always be engaged.

0:30:18.005,0:30:21.265

Yep. No. That’s awesome. That’s awesome.

0:30:21.265,0:30:24.185

So, hey, thank you so much for your time today. So

0:30:24.185,0:30:28.555

Ringo, Thanks everybody who joined. And I think people are now transitioning to the

0:30:28.555,0:30:33.015

session. So if you haven’t it, you can transition to that, but I really do appreciate Ringo

0:30:33.015,0:30:37.055

And this this of course will be recorded and available on demand as well.

0:30:37.055,0:30:41.515

So some of you in fact, might be listening on demand. But thanks very much. Appreciate it.

0:30:41.650,0:30:44.170

Pretty very good. Alright. Thanks, Matt. Thank you.

0:30:49.312,0:30:50.232

Yep. Thanks