Championing Automation: How Quality Leaders Can Drive Organizational Support

Video Transcript

so welcome everyone to our champion automation how quality leaders can drive organizational support Roundtable and really excited because we have three different experts with three different angles of perspective so today we have Chris Huff CTO of rent group who will share how to unline automation with Core Business objectives providing the executive stakeholder kind of perspective so hey Chris good to have you awesome we also have Lee joining us who is the engineering leader at Weight Watches really cool gig and she’s going to provide insights into managing change effectively and also draw from her awesome experience leading a global engineering team so welcome Lee thank you hi everyone great to have you and John John is here he’s Chief Storyteller teller at next it and he’s goingon to offer techniques on Crafting compelling business cases to really Garner further investment in automation so John good to have you thanks for having us awesome so also I just want to tell the audience if you have any questions anytime just jump jump in in the chat or something if it’s relevant I’ll try to interject throughout the round table so let’s get into it I guess the very first question is I don’t know if you’ve had this experience I went to an online on-site event recently and a lot of the vendors told me hey a lot of people are coming here that have no automation whatsoever I was kind of surprised by that so it got me thinking how do you justify maybe the investment in automation over maybe traditional engineering time to Executives um I guess we’ll go around the room really quick we’ll start with John as I see you and then Lee and then Chris so John any thoughts so I mean for me I think the big thing um is is understanding why you’re doing Automation in the first place when you say test automation what what flavor of it are you talking unit testing functional testing security testing smoke testing performance like there’s test automation comes in a number of flavors and it’s really a question of what what you mean when you say Automation and how you want to justify it and I think understanding what you’re going into with the objectives of of this is what I’m trying to accomplish this is what automation is going to bring me this is the value that I am going to bring to the organization with it I think starting there and understanding what that is before you you start trying to justify it um I think that that’s probably the first thing you should consider before you ever even think about a tool or anything else is why would we do this absolutely Le I’m just curious to know from a Global Engineering perspective how do you how do you justify it or how do you get buying from your different teams as well all across organization our position has been uh automation is not always the answer uh if there is a root problem to solve uh where automation would bring that return return of investment then uh we try and see how we can Implement that uh in our organization we’ve been trying to implement the RFC process uh request for comment uh we consult on what the problem is what the solution should be and if it turns out that any type of automation whether it’s integration testing unit testing or UI testing we bring consensus and then uh figure out different approaches and provide a solution is the stance that we’ve been taking recently awesome and uh Chris any thoughts on that that you can add to the conversation around this yeah yeah I have to say John I thought every comment was going to be a story so I was a

little Joe Knows well enough I can do that you want me to and I’m trying to restrain myself no no I’m just messing so I you know look every business especially high growth businesses are all about like speed to market right nimbleness and just being able to respond to instantly changing what seems like instantly changing business needs so this is where I bring automation quality engineering into that conversation so for those of you who are who are old enough to remember unfortunately I’m old enough to remember when muscle cars first got their start right like you had 300 horsepower I had a 350 horsepower cutless Supreme and the seat belt was like tucked up at the top of the door right like it was like this is not supposed to be used in fact I never used it because I W have to fold it back into the into the console so uh you know you had all of this speed but no safety right and so people were dying it was a it was actually a a national kind of emergency at one point to to come up with better regulations airbags all these different things and so when I talk about to leadership quality engineering it’s all about going super fast but doing so in a safe way so that’s from a CTO perspective that’s how I think of automation how I think of quality Engineering in general love it so just remind everyone if you have any questions just drop them in the Q&A I will be monitoring them so based on that once you get Buy in uh I I think you all made the point that necessarily automation isn’t always the answer but also what type of automation are we talking about do we also try to push maybe devops automation or is it always functional automation um do you notice sometime teams miss out on benefits of automation because they’re only focusing maybe on the functional testing aspect of it I I literally had this convers 15 minutes ago um one of the one of the challenges with it is a lot of times people start to equate automation with testing specifically and not really considering the fact that there’s a lot of other things that automation can do for you and bring those Time Savings and safety guards to the table that aren’t the actual testing itself but bring a lot of value in that way and I think that’s something that people need to start remembering is that that it’s not just test automation but it’s Automation in general and you we’re talking about this whole Quality Landscape that allows us to to really be effective um and I think that’s that’s an important factor to be considered yeah and I mean like again just from my my seat it’s a little different vantage point I think in some cases but um I’ve had I’ve had QA teams report directly to me at earlier stages of my career as well and have a healthy respect for the um the discipline I think I guess how I think about automation is you know if I think about the anti pattern I love talking about anti patterns with my team so uh you know indulge me a little bit the anti pattern is you just have a bunch of things that are always at 100% it’s always green or it’s finding the same issues that come up every time and so if you think about signal to noise ratio it’s really not signaling anything it’s just noise for the team every time the automation is just telling you you know basically the story so aside from code coverage one of the biggest things I push the team around is the effectiveness of the automation like how often I mean if it’s not finding bugs then why are we doing it right and so like that’s that’s a like kind of how I think of it and I think is an important aspect of it is to make sure that everything you’re doing in automat in automation is actually adding value and it’s not just testing things that are painfully obvious I I’ll throw in there a failed test is not always a bad thing in fact it’s often times a good thing because it tells you your automation working I mean that’s exactly what you want out of your outcome yeah absolutely if I can speak to a real world example that we are just responding to today uh we think of automation as uh integration with our uh tools also uh tools that we use to monitor crash that is integrated with slack we have a channel where we monitor it alerts when crashes Spike so that type of automation is also important where uh we are responding to real world issues not just responding to automation failures within the organization within our ecosystem but automation that is integrated from outside the world real world that uh informs us of what’s going on and respond to it so that automation is also necessary and uh having checks in place to review look at respond to those types of uh information that come in yeah absolutely great so have the right automation at the right place at the right time I love that so as I mentioned at the beginning you know you all have different perspectives based on pretty much where you are in your organization so I have handcrafted questions for each of you to get your insights the first one is we were kind of teasing John about being a Storyteller but John how do you craft a compelling narrative to Executives I mean investment yeah so one of the things that you know you’ve got to really start to look at is what is the story you’re trying to tell I mean you can’t really craft a narrative if you don’t know what what the actual plot of that narrative is right what what are you actually trying to convey and you know one of the the biggest aspects of it I think and this is why I talk about it all the time and I I’ll harop on it till I’m bleue in the face is really having a good strategy in place for what your automation is going to bring to the table so that you can go into it with a clear picture of okay this is where we’re trying to get to but it’s also really really important to make sure whatever that automation strategy is aligns with the actual organization’s goals and objectives so if you’re not directly contributing to those goals and objectives then your story is going to fall apart right away right there there’s that that twist or reveal that that happens in your traditional story is going to be well the twist here was it’s it’s not actually adding any value so sorry that that didn’t land for and and that’s a really like so many people forget that doing automation for automation sake is not useful right if you can’t articulate why you’re doing Automation and the value that it’s actually bringing you need to stop because you’re just wasting everybody’s time now it may be that you have something that is of value in there but you have to understand what you’re trying to accomplish with that automation first before you’re going to understand do I have anything that’s useful or am I just wasting everybody time and and you know that sounds blunt and brutal but I I went into a place one time and we threw away seven years of automation scripts because they were doing absolutely nothing like they they were running to to Chris’s point we got a whole board full of green check marks and they were doing a heck of an effort to keep those things running over over those seven years but what they were actually get value-wise didn’t mean anything so we just scrapped them threw them away started over so Chris as a CTO how does someone pitch something to you uh is like how do you keep keep informed of what’s happening or do people come up to you and say hey we need to change this here like how did someone pitch you in a compelling way to get your buying to say okay all right go do that yeah I mean it’s it’s generally about solving problems right like you know it unfortunately the I guess the downside of my role is like I end up having to gravitate towards where the problem areas are right the firefighter or just going to dive into that need help um often times it’s not it’s not the space at all but oftentimes it is I I will say one of the I guess one of these emerging well it’s been around for a little while but there’s been this perception as like hey Engineers right right you know automation they can do all of this stuff let’s just let the devs do all of the the quality engineering work and like I I really don’t love that approach sure the developers absolutely can write the code but the the value ad is the people who understand the craft of quality engineering people who understand that discipline and are adding to it and solving business problems with it so I guess the way I look at it and how I’ve managed this with my teams is you’re you’re either just doing QA for QA sake or you’re leaning into that calculus of solving business problems and are creating value through that if you do that you become Irreplaceable terms of like devs are not doing that like I was a developer and I did not like to write my own uh test scripts right so it it’s not something that they want to think about they want to think about their code working they don’t want to have to go in and and think about all these different Edge patterns and stuff like that so um Quality Engineers love that they love breaking code so you have to continue to push the envelope right there’s always this tension of speed performance quality you’ve got to you’ve got to lean into that um that tension and problem solve I I’ll give you one example is um at one of my last jobs we had broke into product teams and we had the the um QA folks that were invol in embedded in the each team and I love that model but um we were we were hitting a bottleneck in terms of of actually writing the test cases like what what are we trying to test because we were building a lot of new Fe features to your point one of the teams came up with uh hey we need to go and Implement specification by example this will be a way we can move fast because we’re actually testing features with customers as we build them and so it was a great way to get the product managers in on the game of actually automating some of this stuff by using the girkin language to actually go and write these things and so that actually became a huge problem solve right we were we were we had all of this kind of I guess you know a little bit of just repetitive work going on right like product managers weren’t enjoying writing them down then the Q or I had to go in and then I got interpret them and basically duplicate that work so that’s the kind of thing that that I love because it creates value not just for the team but for the business as a whole I’ll throw a follow on to yours really quick for Legos that one of the things a lot of people especially developers when they’re testing they they don’t tend to look at it they’re testing to make sure they built it correctly and built it to specification as opposed to does it do what the customer actually needs it to do and the enduser expects it to do and while we think on the surface those two things should line up all the time that is not often reality because the way people like just the mental models of looking at how do I build something versus how do I use something they’re totally different yeah and I love that that that you you use your example because it’s exactly why you you have the disconnect between developers and qas a lot of times because they’re literally looking at it through two different lenses and and their perspectives are just totally different so sorry Lee yeah Le great so how do you keep your teams Global teams I I used to work on a team with eight Sprint teams and that was a disaster you’re Global worldwide like how do you even do it and keep everyone on board make sure that everything’s matching up with your C Business objectives and you know you are getting things done in an efficient manner uh I I’ll answer to that and also as a followup to what Chris said right we’ve gone to a uh place of quality is not just owned by the quality engineering team but quality is everybody’s ownership right so we are not that one team trying to break the code anymore uh from product owners to product managers to engineering managers to the engineering teams everybody owns quality and that’s how we are managing change in our Ever Changing World um you if you are able to uh clearly articulate like John said your root problem and say I am able to solve this with automation then we take that route otherwise we are always uh in the shift left mode how quickly we can get things tested how early we can get things tested uh and continue to iterate on that testing whether it’s automation or manual testing um and if you’ve built up automation Frameworks uh to again John’s Point uh if you worked on it for almost three years and you realize that use case is not working for you anymore that approach is not working for us anymore being uh willing to look at it and say okay I’m ready to change my Approach not married to the idea of I’ve worked on this three years I don’t want to give up on it just yet uh not being in that position but looking at it and saying it is not working how do we change the approach how do we get Buy in from uh engineering teams uh engineering leadership to change the approach evolve the approach so that it continues to work for the business needs um I think those those have been some of our um uh ways in responding to changes that we are bringing in I I’ll tell I’ll tell a great analogy that I love using with people is you know oftentimes they’re like we got this whole this problem it keeps getting worse and worse how do you get out of a hole what’s the fastest way to get out of a hole stop digging right I mean you keep digging it’s just gonna get deeper so just stop like it’s okay it’s a sun cost at this point Let It Go great point so you did mention everyone owns quality uh I I’ve worked maybe at a dysfunctional companies where say everyone owns quality but no one owns quality quality team still owns quality how do you actually get you know because a lot of people see QA as almost like a cost center QA activities as a cost that are not necessarily developers adding to the bottom line how do you change the culture then like how do you get there put processes in place that work right so as simple as uh responding to a production issue uh it used to be that we would hear about it somebody would toss it to QE and we would try to reproduce and respond to it right it’s it’s not that anymore um Customer Center hears about it they bring in both engineering product owner and including the quality engineering team into the conversation and everybody is looking at it together when more eyes are on the problem you have more perspective somebody knows what change could potentially have broken that and somebody could tell that from the QE team that I I tested it in a certain way I missed to test this certain way that’s why you are seeing it in production right so putting in proceed in place identifying this problem and then analyze the root cost bring in the expert to fix it and work together with quality engineering team to put it out there put the fix out there right so it it’s having checks in place implementing the process socializing the process it’s it’s not just happening in somewhere like in a whole deep hole uh it’s it’s out there the engineering team knows how how the process works how QE is involved bringing everybody in uh socializing the process keeping at it right you it’s not you follow that process once a few weeks few months and then forget about it right keep at it keep improving it learn from it make changes so it’s it’s how we adopt the process how we socialize the process is how we bring everybody kind of sounds like you’re talking about setting expectations for everybody such a novel idea and accountability right yes if you don’t have accountability expectations are useless yes absolutely yeah yeah but Chris as a CTO how how do you Jud how do you like um Bal the approach yes you want quality you want quality culture but you know your Management’s coming down and you we need this out the door you know this a money making feature or something like that yeah I mean there’s some there’s definitely some up uplevel leadership issues there that can can lead you down a a bad path right I mean if I if I say hey go build feature a and then the team is working down building feature a it it’s hard to then start to talk in terms of the a common currency around what is the impact of that right then I’ve got to surface up the risks I’ve got to start to bring that to light the way I tend to rate U lead my teams is around outcome orientation right what are the goals we’re looking for because that’s ultimately what I want I want more Maus I want more you know a happier customers so that then levels the playing field and it becomes much easier to have those trade-off conversations because look if I’ve got bugs in production but I’m achieving the goals the customers are happy with the product and the performance of the product and it’s it’s achieving what I needed to do then I’m good like don’t don’t keep go you know I mean you want to keep improving it but I don’t want to um I don’t want to hold up value in effort you know for the sake of something that’s not going to create value so um it’s it creates this this currency that then the QA QE team can then exchange on right because now I can talk about value creation or risk to that value if I’ve got something out there that’s broken and Q has identified that right now the the status of that issue uh especially if it’s impacting kind know critical outcomes uh becomes massively higher and so um that’s generally how I approach it it’s a little harder if you’re more in a in a feature shop where you’re talking in terms of the nouns uh it can get a little bit more difficult uh but I think of um I think of QA teams as really managing risk for the business uh and is is never it’s never a you know all or nothing it’s it’s all about balancing the two yeah I think having having QA people in qes that have a voice and feel like that voice can be heard is very important in an organization especially when you’re asking them to be that that voice of quality or The Gatekeepers to Quality having people that are willing to stand up and use their voice but also having organization that’s willing to listen to those people is is really really important how’s that done John sorry you gonna say something yeah uh and the best way I’ve seen that uh empowerment for the QE org is by owning release management I feel from my perspective it’s not just about uh saying you’re good to go but actually owning the release process along with the engineering team has been really empowering uh to own that value that we get out of and truly understand what we should stop something for and what we can say okay this is fine I know from my experience this is good to be out there I I would I would say to answer your question Joe right along the lines of what Lee was saying one of the best ways to to do that is just taking ownership in general right owning well while quality is everyone’s responsibility at the end of the day somebody has to own it and and take ownership of it and and be that that voice of it and it doesn’t take much to do that just being the person that’s willing to stand up and say this isn’t good enough we have to do something different sometimes that’s enough but you you have to have people that are willing to do that the only way they’re willing to do that is the first time it happens somebody listens to them right if you’re given an excuse and and kind of kind of push to the back and and you for lack of a better word Gaslight them and say no that’s not you’re not seeing reality here this is actually something different we don’t have to worry about that then you’re you’re lessening the chance of that happening in the future and you you even if you don’t understand the reasons why they pull the the plug and say we can’t do this yet or they’re saying this isn’t good enough take the time to understand why listen to them and then come back and have a conversation about it if you have to but give them that voice and allow them to be willing to to pull that emergency stop and say did we have to do something different I think that’s important nice so I guess in order to pull this all together though you need to have some sort of heartbeat how the teams are doing how how things going in production especially when it comes to automation so I know this is kind of tricky topic but metrics a lot of people care about metrics Roi so do you have any metrics for measuring success in automation especially how your team’s doing with this um Chris bman heard from you in a bit so let’s start with you on this one yeah I mean certainly Effectiveness I I was telling you I code coverage and effectiveness of the the automation is certainly uh top of mine um I think and so one of the ways to measure that or I guess the way that I look at measuring that is how often is the automation creating uh bugs creating work to be fixed uh invaluable work to be fixed again if it’s you know telling me the same thing that no one cares about that can be less valuable and are you fixing actual bugs or are you fixing broken tests those are different things yeah and it is something you need to factor into that conversation yeah so uh I I don’t so I don’t um one of a little bit different I guess in my Approach is that I I kind of have what I call uh dashboard metrics and then windshield metrics and so you know for me dashboard metrics are you know like if you’re driving down the road like if you’re if you’re running out of gas you probably need to know about that if your engine’s overheating you probably want to know about that but if you’re staring at the dashboard you’re probably going to run off the road at some point or at least not get to your destination so the windshield metrics to me are you know for for me are always around the consumer the client the the customer facing work that we’re doing and I think you know part of the challenge is that like QA and QE teams get obfuscated from that and that’s that’s a that’s only to the team’s detriment right and so yeah I think as much as possible you guys have like take ownership don’t be afraid to raise your hand and say hey I you know what we want this uh we want this MPS metric as part of our how you grade our team right and so those kind of things are really cool goes back to the first question you asked us around how do I justify the the the cost the value it’s like well hey the last five customer facing issues that were core to this you know NPS metric we were targeting were found by this team so um I definitely you know look at those first and then I have kind of a smaller set that I look at that I call dashboard

Metro awesome Lee any thoughts on how do you measure your teams uh any metrics you use you find helpful I I think uh issues that we catch early early detection of issues one thing as a as a people leader I’m always concerned about is time how much time my quality engineering team can spend on something uh with the world of uh experimenting AB tests uh features uh having multiple variant we just don’t have enough time to test control and variant always so if my automation can help towards finding impact on control while we spend manual testing time on variance then that’s a good metric for me to sort of uh say that I’m saving time I can spend time with my people l in these areas uh time yeah my my people and where they spend their time than I I would say for me I I will well I think a lot of the things that Chris mentioned are super super useful and important for me the one that I care about the most typically is the I don’t care how many we find before we go to production I want to know how many Escape into production and over time does that improve or degrade and that’s that’s what I’m looking at because it’s great that we find stuff before production but I want to make sure we’re finding the things that are going to cause you know loss of Revenue if we get into production with them I want to make sure we’re hitting the right areas before we get into production and so I look for escaped issues things that actually got past the QA process and understand what holes do we need to plug to make sure we don’t miss those kind of things in the future that’s to me my measure of quality is are we getting better at putting out something that is stable consistent and bug free never point though about uh maybe I have seven bugs in there but doesn’t impact value or risk so how do you not use that as like a perverse measurement now like ah you seven that escaped but you’re like yeah but I I think it’s also down to understanding how those bugs tie back to both your your you know revenue and the business critical function of the system are they impacting your users Chris made this point earlier and I think it’s it’s very important that I don’t care if certain types of bugs get out right if we’ve got a font that’s different on this page versus this page or a button is different colored here versus here well are those bugs technically yeah but are they causing Revenue loss or are they causing any you know brand harm no okay that’s not the kind of stuff that I’m talking about I want to classify them but that’s going to be low on my priority list what I’m talking about are those P1 or P Zer issues that get out that cause Revenue loss and cause brand harm or some other business impacting problems those are the things that I care about and all of them should be given some sort of RCA right you want to know what you’re doing what’s getting out why did it get out how did it get out all those things are important but the things that I care about are those p1s and P Zeros that are really like that’s CA that’s costing me money awesome all right we have our first uh user questions from Carlos Hernandez wants to know QA should not only be accountable but also should be seen as a revenue making entity for the organization good best quality will improve Revenue generation bad quality buggy releases will impact Revenue I’d like to hear your thoughts on that anyone have any strong thoughts around this this sentiment

here I I think it’s possible for that to be true but I think it goes back to the very first thing that I said um early on was was have you aligned your quality and automation initiatives to business goals and objectives and the initiatives there where you can actually tie your efforts back to revenue and you know increase decrease whatever but you can actually tie it to actual business outcomes um if you can get if you can say that definitively making it a a global kind of overarching blind statement is really hard because it’s hard to quantify that right you you there there’s no way to prove that but if you’re actually tying it to business objectives and business outcomes absolutely love that yeah and and did Engineers take the time to understand the business um like I said I I kind of managed my team this way and one of the things I’ve observed uh repeatedly is uh developers love to complain and I’m Plumping in QA QE with that with that as well developers love to complain about their lack of empowerment and the one asked me if this was a good feature and then as soon as we moved to this model you still get the same comments like hey you know this was their decision I’m like why you guys should be deciding this together and um it’s interesting they that you know there is a tendency to want to lean back and that means not understanding the business business so you’re you’re nailing exactly what I was talking about earlier that voice and being willing to stand at and say this isn’t good enough like if you believe that say that like put your hand up and say guys this does not make sense yeah and look you know in my career the you know the things that I love the most is if I get a I get in a when I’m going to a team meeting they’re talking about an issue if I get a product manager explaining why we need to pause to go pay down Tech debt I’m like oh man this is awesome and similarly if I hear you know QE leader go those defects don’t care they these these defects don’t matter right like I love that they understand how how it’s used and the impact on metrics to be able to inform me with confidence like that is is uh is just awesome so but is it doesn’t come for free and it’s not I think it’s not it’s not a gravitational pull for a lot of uh engineer including myself when I was personality it’s a personality type that some people aren’t willing to insert themselves into that conversation in that way they don’t like to put that stake in the ground that says you know good or bad um it’s just not something that comes naturally to a lot of people in that field nice Lee any strong thoughts for uh I’m learning uh I was thinking about I was all uh I was thinking about was app reviews in the mobile world how uh how high quality your product is directly corresponds to the app reviews so that’s how QE quality engineering teams can feel that they are part of the bigger picture right their contributions directly contribute to the app reviews well and one of the things I tell especially QA people and people in the the quality space whether you’re doing the development side of it or what not is it and it’s just super true in the mobile space but are you creating something that you yourself would actually want to use because if you’re if you’re willing to let something go out that you would not use you’ve got this need and you would not use this product or you would not like using this product why are you saying that it’s thumbs up good to go like you should at least be raising and highlighting the fact that well it technically works the user experience here is not great guys like that really be thinking about that yeah that’s a great point and I I uh I say I’ll sometimes say this it’s kind of a harsh statement but it’s it gets to the point of what you’re saying John I you know users suffer by what we build right they don’t have to and so like sometimes you just need to sit and read those reviews sometimes they’re over the top but you can feel when you read those you can feel the frustration that that your software created yeah and so uh it’s a good it’s a great Point Lee and I think it’s a it’s good to keep that top of mind I think it’s I I’ve seen so many places where the developers never use the finished product like they never actually use the product to see how it actually works and so even if the QA people are saying that there’s still this disconnect of not like I hear what you’re saying but this is actually how it’s built and this is what this is what like no but you’re not you’re not actually using the product to realize it doesn’t work the way you think that it works the way the users use it is actually this way so yes you may have technically checked every box and built it exactly to specification even in an over engineered way it doesn’t matter but if you can’t actually use it to do what it’s supposed to be for doesn’t matter how well you build they could be the most amazing piece of crafted Artistry you’ve ever built in your entire career and it could completely useless to the end customers love it the chat is blowing up S says great Point uh Jason says eat your own dog food exclamation point I love it so is this part of a I I always think uh like getting the user experience being uh aware of how your application is actually being used and perceived in the wild uh do you have any suggestions on maybe quickly uh like maturity models or road maps for folks that maybe are on different parts of the journey with automation know I when I spoke speak to vendors a lot of times they either have Super Hyper automation users or you have people that haven’t done it at all like do you have some sort of mental model you can give folks to help them on their road to automation awesome I feel like you set me up for this question yeah I

did for exactly the reason you just talked about we I have over the years kind of started to put together this very high level mostly targeted at leadership and and higher level organ organizations to kind of understand where you start from a QA standpoint and like where you’re trying to get to and it goes from you know you don’t have any QA or testing to manual QA you got some automation you’re doing a little bit more automation it’s kind of a 50-50 split now now you got some cicd stuff going on and then finally you’re in a full cicd you know continuous delivery continuous deployment model like that’s great but in the mix of that there’s seven different phases you have to go through and each one of those have their own own life cycle of of success and failure that you kind of have to work through understanding where you’re at in there and that that the end of that stage does not mean the end of your journey but you’ve got more that you can do to help the organization um whether it’s you know the organization as a whole or just your team um understanding that that there is a Continuum there it’s not a black and white answer Le how how do you make sure all all your teams and your organizations are up to the same level level I assume different teams will be at different levels yeah is that something you you monitor or you have to keep I mean you part of it is having again some sort of strategy in place to set goals and objectives for the organization and being on different levels may be entirely okay right it it may be that some teams are more advanced than others and they’re ready to move on to different levels but having something that is kind of your Guiding Light or north star to keep you on track regardless of what team you’re on that’s important because otherwise you’re just going to flounder and and you’re going to say Well they’re doing this so why aren’t you guys doing that well because we’re not there yet we’re not ready for that we don’t have the skills for that or the product’s not to that stage yet you you got to have something that can give that some some context and I think that’s important y Le any thoughts like John said I I was going to say it’s all right to be at different levels the what would matter is being one engineer ing team you can have pillars domains within the engineering team but being one engineering team having some sort of standardized process um then you can fit people into whether manual testing is needed some people might be really doing well with shift left process some people like personality right some people might not far into the right spectrum and they’re they’re only in production baby that’s it that’s all we’re doing yes so so some some form of standard process that that solves the why the root problem being one uh and another thing we’ve uh experienced is not making decisions in a silo as a QE organization yeah that helps also you can be at different levels but don’t make decisions on text tacks or processes yeah in Silo and Chris oh I yeah I mean these these experts here I agree with all that love it all right so we have three minutes left so let’s uh uh random question AI overhyped underhyped or properly hyped and uh Le what do you think on AI and how it maybe can help QA QE I’m I’m uh in personally in research mode my team is in research mode we are we are looking at how it can help uh we are slowly starting to get in uh in that area but at this point very much research mode can’t say whether it’s overhyped or rightly not not yet yeah uh so I think it was overhyped for a long long long long long long time and so um now it’s starting to deliver it feels like a step function change I don’t think that we’ve seen the step function change yet I think there um like with anything right like the the capabil is there but now the solutions that actually meet consumers needs need to be built um and I think that’s that’s happening in small steps but I think we haven’t seen a step function change just yet I one one point on this one I think um you know there’s a a couple you know with anything whether when things commoditize or automate you’re you can either fight it like you you push back against it or you embrace it to create value and the teams that embrace it to create value are the teams that just went out right like if you fight it you’re GNA get displaced by it if you’re like oh great I don’t have to do this anymore like my teams are using um co- co-pilot for um unit tests and some even some other kind of broader string tests and stuff but it’s um and so like the attitude should be awesome how do I shove that thing that was adding value that I was doing into this this uh automated realm or this um you know kind of you know uh lower level realm so that I can do higher order work because you always if you if you’re layering in higher level work then you can do you know kind of what we were talking about at the beginning you can create value and demonstrate that value more otherwise you’re going to just be seen as being displaced by this technology so I I to me I think it’s to Chris’s point it was over overhype for a long time and now that it’s here there’s a lot of things that you can do and get value of it you know I challenge organizations that don’t have QA leadership to use it as you know how do I do QA better it’s actually quite good at that um how do you how do I create Automation and actually help you build that Automation and help non-technical people quickly start to contribute I think those things are incredibly invaluable and like Chris said the people that embrace it are going to be successful and the people that fight it will likely up finding out that they should have embraced it awesome thank you John thank you Lee thank you Chris for joining us today thank you Katon for putting this together and this excellent event and for having us today to cover these important topics so thank you everyone for joining us and hopefully we’ll see you here next time as always test everything keep the good cheers