Measuring Software Value and Trust: Let’s RAVE

Shannon Lietz has helped pioneer and codify much of the work that’s become known as DevSecOps. Over the last decade, she has continually sought how to better bridge the worlds of information security with what actually needs to be secured and who needs to be doing the securing. Shannon is currently VP of security at Adobe, which is the fifth largest software company by market capitalization. 

In this transcript of her 2022 DevOps Enterprise Summit presentation, she shows that even though the term DevSecOps is being widely used in our industry, we still have a long way to go to truly create the ideal interaction between information security and the rest of the organization. In this presentation, Shannon shares what she’s doing right now with this community. Specifically, her work with RAVE metrics. 

___

I’m really excited to be here at DevOps Enterprise Summit. It’s been a lifelong dream. I am here to share a little bit about my recent research and current research. As you all may know, I have been in this industry for a long, long time and I’m going to share with you today a little bit about RAVE metrics. So let’s RAVE. 

We’re going to measure software value and trust and talk a little bit about the overview of some of the work that I’ve been doing to pull it together. And then I have an ask for you at the end. So stay tuned. 

I’ve been in the industry so long that, honestly, I started out as a Dev. So as much as I’m a “security professional,” I would say I’ve just really become a DevSecOps professional over the course of my career. 

DevSecOps

So maybe you’ve heard about DevSecOps. I always ask people, have you heard of it? And what I find over the years is it went from a conversation at an ISACA presentation (you may not know what that is, but basically it’s a group out there that does information security and it’s a practitioner’s group) and I remember my first slides. I actually had a tortoise and a hare and I was explaining how the security industry needed to evolve. And so as much as folks here are really involved in DevOps, I believe that a lot of us are really trying to bring security into the forefront of software and make it much easier as well. 

So I’m sorry about DevSecOps, because the name does have a whole bunch of folks in DevOps kind of worried about that. But at the same time, I’m also not sorry because I am finding that over the course of the last 10 years, software has been increasingly getting better at doing security before it gets released.

And that means that we’re checking our software and making sure that it’s as resilient as possible. So that’s my plug for security. I actually have an interesting take on what I think has been happening in the security industry and DevOps and all these things. And so this talk is really going to be focused on more about measurement and some of the things I’ve learned. 

But I’m going to start you off with sort of the journey toward how I get to this point with this software metrics conversation? If you look back in 2014, the essence of all of this came from sitting down and realizing we were security folks and the developers were basically moving around some of those clunky clipboards that we had. And so we finally sat down and wrote down really what we needed to do to change. And it was like we need to lean in instead of say no.

I think at one point I got a T-shirt sent to me that said the word no on it with a period. And it was like that was the brand that was being sent around, that was what security was really all about, it had to be our way or the highway. 

In software development, that can really be a pain, if you will. Being a developer myself, I would say I know what it’s like to be sitting there trying to make something work and then being told, oh, by the way, you have to layer on all these different things to be able to make your software secure, capable, available at the same time that you’re just trying to figure out how to make it work for somebody and make it useful. 

So in 2014, I would say we kind of put this down, and it was a little bit of an experiment and at the same time it was a pretty good experiment, I think.

Now what was also really interesting is in 2014 we knew we needed to get data and security science. It was actually pretty expensive back then to try and do what people are doing these days with security capabilities and security science, if you will. Big data was pretty expensive. It’s actually gotten way cheaper. It’s become more commoditized over the years. 

And then this notion of business-driven security scores really getting to the point where we weren’t just saying, here’s a list of things you have to do, thousands of things you have to do, but really getting to the point where how can we summarize it and make it much more simple? And so I would say what came out of that 2014 realization has been many, many years, but one of the big most critical things is we’ve been putting all of the security capabilities into the CI/CD program.

We’ve been actually building things into the software pipeline. And at the end of the day, I think it’s been measurement that’s been the most critical and most necessary element of pulling people together. So why do I say that? And by the way, I find it to be extremely challenging to talk about security measurement and how you’re going to actually be part of the conversation of building software.

Metrics

So when it comes down to it, the reason behind this is metrics really create a little bit of that guiding light, that North Star, Measure What Matters, the John Doerr book, how do we actually start to determine what we’re going to do and how we’re going to solve problems? And so at the forefront for a security professional, we’re all trying to figure out how do we actually make this thing that’s really important, getting rid of adversaries in our software, taking away their advantage, taking away the attack surface, how do we really make that come to the forefront? And the idea behind this is that we could go focus on what we can measure.

The idea behind metrics is really interesting and unique, and what we can measure is often not what we should measure. It’s not necessarily what matters most, but it’s got numbers associated with it and they’re actually something that’s relevant so it can help the conversation. And what I find is that commonly those metrics are actually around value creation. 

So there’s been a lot of amazing work out there. So don’t take any of this as being pejorative toward the work that’s been done out there because it actually isn’t. In fact, I’m a big fan of the work that’s been done out there. I just have a different take on how I think it needs to come together to truly take us to that next level of how do we actually all work together to inspire a community of software development that I think is at the forefront of what people thought they were putting together when we built out things like software as an industry.

So we have a ton of stuff out there, the DORA metrics, the QUANTS metrics, the SRE metrics really all centered around this notion of developer productivity. And some would say as a developer, the last thing I want is for somebody to measure how I’m doing in terms of productivity. And then others are really keen on it. 

We really need to get to the point where we’re measuring out our wellness, how we’re actually doing, what we’re doing. Are we getting the help we need to be able to build software more complete, if you will? 

And over the years, I’ve also heard that a developer has to do everything. They have to be able to make business decisions, they have to be able to make security decisions, they have to be able to make all these decisions, and they have to be able to write the software at the same time.

And maybe that context is something we need to change. That it’s not necessarily that they have to do all these things. It’s that as a community, we all have to do enough software development ourselves. So this notion of maybe being part of the security community, but creating simplified capabilities that could be included in software is something that is part of the paved road path of where a developer can be productive and solve for customer problems. 

So, long while back, I started out trying to map out what did I mean by DevSecOps and what did we mean as a community about what it was going to do to fit into the process. And at the time there was a little bit of work done out there around software development, lifecycles and security folks were showing up and trying to leverage their tools during whatever part of the process.

So hey, I need access to GitHub to be able to look at your code and then come talk to you about how that might be working. And ultimately, value and trust was something that I think came to the forefront for me is it wasn’t that we just needed to do this notion of measuring value creation, but trust. And so I love this picture because it’s really about if you have value and trust being measured, this notion of software being delightful is something that I think comes to the forefront. So I took this picture to heart. I actually am a big fan of some of the products we have and one of the ones that I spent some time on is stock. I do tend to love to try and visualize and narrate how I’m thinking and feeling about the journey that I’m funding with my research and my time.

And so I did learn a lot at Intuit previously where we talked about things like design for delight and having the principles of designing for delight. And really that customer understanding of software that delight that’s being created is I think the driving force that we’re all trying to measure. But by the way, measuring that is hard because measuring trust is hard and measuring security is hard, measuring all these things that are a little bit more, I think not necessarily part of just taking code and putting it down and putting it onto a system. And so today I’m going to talk to you a little bit about how we actually bring those all together.

And another one that has recently inspired me has been Scott Belsky who works at Adobe. And so if you take Scott Cook and Scott Belsky, I have definitely been inspired by both of these gentlemen to really think past the notion of what we do day to day and more about how do we actually take this as a community, a journey together, and how do we really bring this complexity of software to the forefront where we could solve human problems and simplify that for our customers? So what does it all result into? Well, if you hunt down both of the inspirational folks and you look at the basis of their work, some of it is really about what makes someone a net promoter?

I mean, ultimately if you think about it, what products do you leverage? Which ones make you happy? What are you buying out there? What products do you have sitting in front of you? Those are things that if they were to go away, I think somebody mentioned to me that Superhuman said that if their stuff was to go away, they actually measure the notion of would you be disappointed? And I think it’s a really good measure because it helps to understand a little bit about that trust algorithm that we’re talking about here. So what makes someone rave about your products to their friends, meaning what makes them put their credibility on the line and really talk about your software in terms of it’s the most amazing thing.

It does this, it does that. And oh, by the way, it’s always up. It’s always online. That’s how we got to the cloud in the first place is we were just unhappy with, hey, this system isn’t scaling, there’s error rates associated with it. And as we all research really what the expectation is of the customer, we invent all these amazing capabilities that make it so we can get to that potential of what people dream about in terms of using software products. I think that’s the most amazing clearest vision I’ve ever had about why software is something that speaks to my heart. And so to me, I wanted to sit down and really say, all right, what is that algorithm that makes me rave? And all of a sudden it was like I put two and two together. I’m like, well, maybe there’s something here that’ll make me remember all these metrics and have a harness for being able to do so.

RAVE – Measuring Value and Trust

So I’m going to talk to you a little bit about the four things that I think are important for this trust algorithm. And one of them is, as a customer, I can rely on your product because it’s resilient. And so resilience is an interesting part of this journey. I picked out that word very specifically. I didn’t talk about availability, I didn’t talk about some of the reliability, securability as being that forefront, but for the customer really having resilient software, that rugged software if you will, it’s something that I think they deeply care about because it is part of that algorithm of why they come back for more. Another one is, why do I want to do things with your software? So having the ability to say, I want to adopt your product because it helps me. I think software has to help you for you to want to use it and to value it. So the utilization of software is a really critical element.

And then another one is your product has high-velocity feature evolution, meaning it’s changing and evolving and it’s taking my feedback and I’m getting more out of it over time, so you’re earning my trust continuously from this part of the algorithm. And then the last one is like your product is a low error rate. I’m not disturbed by constant interrupts. When I click on something, it’s not going to an error page all the time. I’m actually seeing that you put some time and effort into the experience and that your error rate is actually pretty low. 

So I call it RAVE. I bring all these measures together. I like to build upon what other people are doing, not just reinvent the wheel. I coined this phrase because I think that branding, bringing something together and really focusing on the most important elements of what we’re all trying to do is how we’re going to get to that point where we’re all talking about it in a way that makes it forefront and we’re going to solve that big problem, which is humans have better software to solve problems that we all need to solve.

So what does it all mean? 

So RAVE is really resilience, adoption, velocity and errors. Now, I’m not going to predict every single metric that’s going to be at the forefront, but I am going to try and get us to start talking about and debating which metrics matter most in the community. 

So using the RAVE framework, what I’m really trying to get out there is that we’re bringing not only developer productivity to the forefront, but we’re also adding in this notion of customer delight.

Resilience

So let me talk to you real quick about the resilience that’s out there. Software’s designed gracefully and it helps us to establish an operation with thresholds. If you think about it, resilience is really about saying what you would like to achieve and then building your stuff so that it eventually achieves that outcome.

I believe thresholds are critical in establishing good resilience. And the way you do that is really to look at the metrics associated with it that are out there. So are you willing to put down what your service level objectives are? Do you have availability with five nines? I think that that’s a threshold. It’s like what is your threshold? If your software is down 1% of the time, did you meet your threshold? Is that the risk you’d like to take for your company? I’ve got this notion of securability and many people have heard me talk about it throughout the years. And if you don’t know what it is, definitely look up securability, look up my name and go take a look at what I’m driving in terms of trying to make security measurement come to the forefront because I do think it’s critical. And there I think five nines would be probably pretty drastic because there are things we have to do to be able to balance security against all the other resilience metrics.

So I do think that you could say, Hey, is it five nines or one? In many cases, if you look across the industry, adversaries are becoming much more aggressive and holding them back and creating great software is about making sure the risk is dealt with in the right way. These are really important. If you look, DORA is mentioned in here, so I’m giving you an idea of metrics that you could put into your resilience area. I’m not telling you which one to go chase down, but I do know that when you focus on the most critical things of the resilience problem, where are you having the biggest challenges, then you tend to put priority where it matters most. And so I would say the balancing of the ilities, we’ve all talked about that in the past as well. The balancing of the ilities is really about these thresholds and how you actually focus on the most important parts of what you’re developing in software.

Adoption

Software is able to be actively adopted and fully utilized. It’s one of the biggest things we all want in the industry is not only am I building it, but you’re using it and you’re getting a lot of value out of it, is the way I like to talk about it. So some of this is coming through with things like product-led growth and I think monthly active users, that MAU that’s out there that’s coming to the forefront, TP99, customer set. Product-led growth is at the forefront of really helping us to think about software is only useful when it’s being used and when it’s being adopted. And when people are delighted, they really do talk about and rave about your software. So I do think that this is part of that algorithm as well.

Velocity

Software achieves the rate of change that matches the expectations of its community.

I think when we talk about developer productivity, there’s a really big sweet spot here in velocity that’s helping us to understand that customers truly do want us to evolve and take their feedback and change and do what’s necessary to continue to create value. It’s not just that they adopted your software, it’s that they actually will stay because you continually change. And so here you can see the SPACE metrics, the product-led growth metrics and DORA metrics are pretty forefront. The one that you choose or the two that you choose or the 10 that you choose, it’s really up to you how you want to manage the velocity part of this algorithm for trust. But I do think that it’s a critical element and that’s why, like I said, I’m a big fan, so it’s not an “or” in my mind it’s a “and.” And in a lot of ways I actually believe that there’s so many great metrics out there that can really help all of us to have the right debate and learn from each other in this space.

Errors

And then finally, errors. Software is successful in achieving quality targets when it has a low number of introduced errors. No one wants to use buggy software. It’s one of the reasons why we all have things like bug tracking software out there because errors actually erode the experience. And so some of the great metrics that are out there, like error rates, I love the one error budgets. In fact, there was somebody talking about security budgets at one point following the SRE metrics that are out there. Defect leakage is important and test case pass rates. And so you’re going to see, I think if you were to look out there that there’s some amazing metrics in each one of these categories that are part of RAVE.

RAVE Framework in Reality

Okay, so what does this really all look like in reality, and why does it matter to me as a security professional and the rest of us as a security community?

Well, the truth is, is that security needs to be part of the conversation and part of these critical metrics. And as you saw I said something about securability and resilience. Well, it was through all this inspiration of understanding what was truly out there in terms of the total population of metrics that changed my mind about it just being securability. That was my driver. And in fact, security has to be part of this conversation in a meaningful way. So if you think about it, that means securability. Part of resilience here means that we have to have our software be malware resistant and it means we have to have it adversary resistant. We need to be able to focus on things like adoption and realize that in some cases the adoption may also show adversaries that are trying to adopt products that we don’t necessarily want to be attracting.

Velocity, we really do need to make it so that developers have the tools they need, the capabilities they need built into the CICD. And part of that conversation is we should be focusing the security professionals on the velocity that developer velocity as part of the journey as well. And finally, with error rate, I know a lot of security professionals spend their time on errors and defects and we call them vulnerabilities and whatnot, but truly we all care about the same things as a community, whether we’re security or dev or ops. And this is a unifying force. In the absence of utilizing this type of framework for building trust, I think we’re really going to be missing out for many, many decades to come. And I do think this does solve a couple of things, which is bringing security to the conversation without it essentially we’ll still be silos.

We’ll still be having challenges. And I think that’s where we have to change. And like I said, I’m inspired to get to this journey as a security professional because really I learned about creating customer delight and I think security is part of that conversation of creating customer delight. So for me, it’s all about creating the right experience.

Now Gene asked me to put this slide in and I’m really excited about it. So the help I’m looking for in this is I really want to have you help me take this RAVE metrics survey. There’s going to be some questions in there about dev and security, and I do think that that data would be at the forefront of helping us to bring this RAVE panel together and create trust in software.

Also, if you could reach out to me if you’re interested in debating with me about metrics, the good ones, the bad ones, the why, and want to do a podcast with me, please reach out. I’m totally interested. I’m going to be doing this in 2023 and something new to look forward to. With that all said, thank you so much for your time and let’s get back to DevOps Enterprise.

Let’s RAVE as a community and bring measurement to software value and trust.

- About The Authors

Shannon Lietz

Information Security Executive

Follow Shannon on Social Media

•