Gene Kim (00:00:00): Welcome back to The Idealcast. I'm your host, Gene Kim. In the last episode, I had such a fun time interviewing two of my four DevOps Handbook co-authors, Patrick Debois and John Willis. So today we have on my other two co-authors, Jez Humble and Dr. Nicole Forsgren. As I mentioned last time, the original idea was to interview each one of these co-authors and ask them four questions. One, tell me the story behind your original DevOps "Aha" moment, each of which were written about in The DevOps Handbook. Two, what is the most interesting thing that you've learned since the book came out in 2016? Three, what is your favorite DevOps pattern or practice? And four, what is your favorite DevOps case study? So that was it. I asked each one of the co-authors those questions and, holy cow, those questions took each one of those interviews into incredibly surprising and unexpected and super fun places. (00:00:55): I had mentioned last time that they were especially surprising since I had spent literally hundreds of hours with each one of them and never heard those stories, and that's why we broke them up into two episodes. (00:01:06): So just a few words on The DevOps Handbook. Last year, I read through the first edition again, which we released in 2016 and my reaction was, "Holy cow, I really love this book." I think the book stands up so well after six years in a way that so many technology books don't. I think it's because we did a good job sticking with principles, which should remain, if not timeless, they should remain relevant for a decade or more, which is a characteristic of a great book. And since the book came out, it has sold over a quarter million copies. (00:01:37): But as much as I love the first edition of the book, I love the second edition even more. There's 15 new case studies, mostly from the DevOps enterprise community, including Fannie Mae, American Airlines, the US Air Force, Adidas. There's over a hundred pages of new or updated content, including so many of the solidified learnings from the State of DevOps research and the Accelerate book. There's a new foreword and material from Dr. Nicole Forsgren. There's an updated afterword, including sections from each one of the five co-authors. So again, I really want to thank Dr. Nicole Forsgren for leading this effort. (00:02:12): To fully motivate this, forgive me for repeating a story that I mentioned last time. Like many authors, I find the prospect of doing a second edition of a book to be a very challenging endeavor and I know it's not just me. I mentioned last time that one of my favorite interviews of an author is from Nobel Laureate, Dr. Richard Thaler, who wrote about his pioneering work in behavioral economics in a book called Nudge. He was recently on an NPR Planet Money podcast, and he talks about how important it was to him that the words "Final Edition" appear in the title because of the huge undertaking it involved. So when interviewer Greg Rosalsky says at the end of the interview that he looks forward to interviewing Dr. Thaler in 13 years, when there's a new edition of the book, Dr. Thaler responds, "Go to hell, Greg." What a fabulous, fabulous interview. I will include a link to that interview in the show notes. So thank you to Dr. Nicole Forsgren for making this second edition possible. (00:03:10): So in this episode, in my interview of Jez Humble, we'll learn about some projects that Jez and I worked on together before The DevOps Handbook came out, what life is like for him being a Site Reliability Engineer at Google, supporting, among other things, the Google Cloud Run product, which I love so much and what he's learned as an SRE. The story behind his DevOps "Aha" moment in 2004, working on a large software project involving 70 developers. They had been working for years when Jez joined and every time they deployed, every couple of months, it would take them an entire weekend involving Gantt charts and multiple teams. (00:03:47): The amazing architectural properties of his favorite programming language, PHP, and what it has in common with ASP.NET, this amazing virtuous property that are missing from other programming languages and platforms and the importance of being able to get fast feedback while you're building something. The anguish that he felt when Mike Nygard's amazing book Release It! came out, wondering if there was still a need for the book he was working on, which was Continuous Delivery. And Testing on the Toilet and other great structures for creating distributed learning across an organization and why this is so important to create a genuine learning dynamic. (00:04:25): And after that, in my interview with Dr. Nicole Forsgren, we will talk about what she's working on now as Partner and VP of Research and Strategy at Microsoft, some of Dr. Forsgren's goals as we work together on the State of DevOps research and how it feels to have those findings so widely used and cited within the technology community. About the importance of finding the link between technology performance and organizational performance and why it probably was so elusive for at least 40 years in the research community. (00:04:56): What she's learned about the importance of culture, how it can make or break an organization. And the importance of great leadership and why the Dr. Chris Strear story about improving flow in an emergency department in a hospital, and the bonus story from the U.S. Navy are her favorite case studies in The DevOps Handbook. (00:05:16): Okay, let's go to the interviews and I hope you enjoy these conversations as much as I did. You're listening to the Idealcast with Gene Kim, brought to you by IT Revolution. I met Jez Humble in 2011 at a Kanban for Operations event that our mutual friend Dominica DeGrandis ran and it is difficult for me to overstate just how much I've learned in almost every interaction that we've had. I had read his book Continuous Delivery before, but the working calls that we had for over four years, working on The DevOps Handbook from 2013 to 2016 showed just how much I hadn't fully internalized and just how radically different these newer ways of working were, for both developers and for operations. During those calls, Jez continually shocked me with stories about how, say, InView handled database schema changes, how Facebook pulled off their dark launches, how you actually do trunk-based development, how you do blue-green deployments for thick client point of sale devices, and so much more. (00:06:24): Jez and I also worked together on the State of DevOps Research Project in 2013. This was one year before Dr. Nicole Forsgren joined the team and I still remember how it felt when we found those first clusters of high, medium, and low performing organizations and how exhilarating that was. You know, when I tell a story like this, it sounds like Jez and I had identical theories of what was required from dev and ops to get high performance, but especially going into that first year of the study, we had some areas that we definitely had different views on and we were using the study to see which one of us was right. (00:07:01): So one of the biggest bets that we had was who ideally should be performing production deployments. Representing the side of development was Jez, which makes sense, given all the experience that he had doing continuous delivery. He believed that there were so many idiosyncrasies in the code that only developers would be able to make informed judgements about how and when to deploy the code. Representing operations was me, which I suppose also makes sense, given the fact that a book that I co-authored in 2004 was titled Visible Ops: Implementing ITIL in 4 Practical Steps. I believed that the production environment had so many idiosyncrasies that it was only operations people who could make informed judgements about how and when to deploy production code. (00:07:45): So he put a question into the survey, "Who performed the production deployments", "Dev", Ops", or I suppose, "Other", and we had a bet about who would have the lower change failure rates. And the answer was pretty surprising to both of us because it turns out it didn't matter at all. If you take the entire population where devs deployed and the entire population where ops deployed, and you compared the change failure rates, it was basically the same. It turns out it didn't actually matter who pushed the button. What did matter was what technical practices were they using, to what extent were they using version control for all artifacts, both for code and the environment, to what extent were they using automated testing, to what extent environment creation was automated. And to what extent was there an automated way to deploy code into production. (00:08:34): It was those things that mattered. And if those things were there, it didn't matter who pushed the button to deploy code. So during the State of DevOps research, there were so many other examples of similar "Aha" moments and personal theories and bets that we had made that led to such surprising outcomes. And in this interview, Jez also reminded me that we worked on another project together just around this time. (00:08:55): In 2013, Jez had created a conference called FlowCon. Its goal was to span boundaries. He wrote, "We believe that the best products are created collaboratively by people with a range of skills and experiences." So he put together a programming committee that had representation from UX, testing, operations, product development, and development. So with me on that program committee was Jez, Elisabeth Hendrickson, who you heard from in the first season, Lane Halley and John Esser. (00:09:27): I learned so much about how to put together amazing conference programs through this experience and I formed so many of my own sensibilities of which I talked about at length in the last episode with Patrick Debois and John Willis. And here's another thing that came out of those nearly five years of having nearly weekly calls with Jez on one project or another. I've mentioned that after learning the Clojure programming language in 2016, I went from two decades of self-identifying as an ops person to now decisively self-identifying as a developer. (00:09:58): Looking back, I think what set the stage for that was these calls with Jez which significantly upped my own expectations of what anyone should be able to do for themselves. In fact, in this interview, you're going to hear about how the survey engine that powered nearly all years of the State of DevOps Report projects were run on something that Jez built in PHP. (00:10:19): So these days, Jez is a Site Reliability Engineer at Google supporting Google Cloud Run, my favorite way of running almost everything in production these days, and a bunch of other customer-facing Google services.I asked him about what he's working on these days. (00:10:37): So I've introduced you, Jez, and I've gushed to you about how much fun and how it's rewarding to have worked with you over the years and more adventures to come, certainly. So what are you working on these days? Jez Humble (00:10:47): Yeah, so I just want to, first of all, echo what an absolute pleasure it's been to work with you over the years on State of DevOps, on The DevOps Handbook, on FlowCon, in fact, back in the day, and then- Gene Kim (00:10:58): That's right. Which is where we met Randy Shoup, who built Google App Engine that we were talking about before. Jez Humble (00:11:07): Yeah, exactly. He was a big piece of App Engine. What I'm working on now is I'm an SRE at Google. So I am an SRE as part of the Serverless Team. So we are responsible for App Engine, Cloud Run, Cloud Functions. I'm on call for those products, so try not to break them. And yeah, having a lot of fun learning how Google works, how Google does things and how SRE works. Gene Kim (00:11:35): Is SRE everything that you hoped it would be? What's been the best and the worst? And by the way, I actually love Google Cloud Run. I think it's just brilliantly conceived. Jez Humble (00:11:45): Yeah, it's great. People love it. I think it definitely filled a gap and getting more and more popular, it's seeing huge uptake. And it seems to solve a real problem that people have and people like using it, as you say, which is great. So yeah, I'm on call for Cloud Run and also for App Engine and Cloud Functions, as well. Gene Kim (00:12:09): App Engine. Jez Humble (00:12:09): Yeah, it's funny, people talk about Heroku, but App Engine was there first. And I think it enabled so many things that we talk about today as if they were obvious, but it never seemed to get the recognition that, say, Heroku did. And Cloud Run, I think the paradigm is just much simpler and it's much easier to understand and the life cycle's easier and it's more flexible. And we are doing a lot of active development on it. There's new stuff coming out all the time. So yeah, glad that you like it. Gene Kim (00:12:47): I do. Jez Humble (00:12:48): And I really like working in SRE. It's been an incredibly steep learning curve just because the systems that we operate are so large and so complex, but, my God, I've learned so much and it's been fascinating. Obviously Google invented SRE, so there's a lot to learn, but honestly I've really enjoyed just focusing on my job and not doing anything else, hanging out at home. (00:13:13): I don't give talks. I don't really do any extracurricular stuff anymore. I'm just... I've got my job and I've got my home life- Gene Kim (00:13:23): Life. Jez Humble (00:13:23): .... my work from home. And that's really nice. My life was pretty complex for the 10 years before that. And now it's pretty simple and I can just be very heads-down focusing on what I'm doing. And I'm learning a ton. And it's really interesting. And working with people I like. Gene Kim (00:13:39): Give me one example of something fun you learned that was rewarding, mind blowing. Jez Humble (00:13:44): All right, so it's kind of interesting. SREcon... What I'm working on at the moment, SREcon a few weeks ago, Narayan Desai, who's a colleague of mine working in SRE, but for some of our data products, he's come up with this way of thinking about reliability that is based on basically taking your workloads and then profiling them statistically. You find cohorts that have low variability in the distribution of whatever it is you're interested in, which in my case at the moment is latency. So you take your entire workload, divide it into cohorts that behave similarly that have low variability. And then you kind of look for this property called stationarity, which is that the statistical distribution stays the same over time. And you look for ways in which the workloads divert from that. (00:14:36): So I really recommend checking out his talk, but it's basically an alternative approach to using thresholds monitoring. And even SLOs fundamentally are threshold-based, but they don't take into account what loads are important versus not important, which kind of workloads are behaving in which ways. They don't give you an understanding of what's going on under the hood. And so I think this is really promising and really fascinating. Gene Kim (00:15:00): Is it just another access of behavior of a workload, besides error rates or so forth, as to expected- Jez Humble (00:15:08): So instead of setting a threshold for error rates, and once the aggregate error rates go below the threshold you get an alert, we're looking at, "Well, let's look at the actual workloads and see how the workloads are changing their behavior and alert based on changes of behavior from the statistical distribution that we've modeled for those workloads." (00:15:29): So you don't have to set a threshold. You just say, "Well, let's look and see if these workloads have changed, how they're behaving." And that is based on the actual behavior of the system rather than some arbitrary threshold. And it gives you a lot more insight into what's actually going on in your system. So I think it's very, very interesting, fruitful approach. Early days yet, but I'm finding it pretty exciting. Gene Kim (00:15:54): That's exciting. I'm so happy for you, Jez. (00:15:56): Gene here. It was at this point when I was reading the introduction that I had written for him, and I was telling him how fun it was, summoning up all those memories and what it felt like for me during those endless DevOps Handbook writing sessions. And at times how bewildering they were to me. (00:16:12): Summoning up those memories, I just remember how exhilarating that was. And I remember how many times I asked you, "Tell me again the dark launch story, because I don't fully get it. I don't understand what the big deal was. And just the Aha moment." I tell you this to convey my gratitude and just how much fun it was. Jez Humble (00:16:28): Not at all. And I shared those very strong memories of learning so much about how to think about high performance. And before Nicole joined the team, you were the stats expert. And so you were the person who came up with those diagrams and yeah, that was very cool to see. So yeah, I shared the fact that it's been an incredible collaboration, very exciting, very fruitful. And obviously The DevOps Handbook has been incredibly impactful. Gene Kim (00:17:02): So Jez, you had introduced to me all the exciting work that was happening as you were working on the Continuous Delivery book, presumably 20... When did you start the book? Jez Humble (00:17:11): So the book started 2006. Gene Kim (00:17:13): 2006. Okay. So a gestation period much like The DevOps Handbook. Jez Humble (00:17:20): Yeah, it was a four year project. Gene Kim (00:17:22): Ah, brilliant. So all this work was convergent together, everything from the continuous integration community, the extreme programming community, continuous delivery, Kent Beck, so much more. So can you describe what it felt like as you saw all these practices coming together and how that informed you writing the Continuous Delivery book during those four years? Jez Humble (00:17:42): Yeah, that's a great question and I could probably talk a lot about that. So where it started was at Thoughtworks where I got a job 2004. And one of the first projects I got put on was the 70 person project that I've described before at a large Internet provider in London, where we were building a replacement system to replace their existing system, which was all written in PEARL. Their architects had gone and bought a bunch of WebLogic licenses so they had to rebuild the system in Java so they could make use of these WebLogic licenses they'd bought. And we ended up working on this team, actually getting the software to deploy into a production-like environment, which was Solaris, and with a bunch of automation that devs had written that the ops people didn't like, because it was written in Ant, and they wanted to use Bash. (00:18:32): And so I wasn't aware of being part of a movement at all. What I was aware of was coming from a previous situation where I worked at startups, where there was two or three people on the technical team. And we would just FTP directly from our work stations into prod multiple times a day. There was no automated testing. You just check it works in your workstation, FTP it. And then if there was a problem, you just fix it and FTP the fix. Gene Kim (00:19:00): And what were you FTPing? JAR files? Jez Humble (00:19:03): ASP.NET. Gene Kim (00:19:04): Yeah, okay. Jez Humble (00:19:06): So it was mainly Visual Basic and MS SQL Server on the back ends. And yeah, we wouldn't even deploy the whole thing at once. We'd just deploy whatever follow had changed. And yeah, we messed it up a few times and broke things and there was probably things that were broken that we didn't notice. So it wasn't exactly a high quality process, but it worked and it was fine. Gene Kim (00:19:31): And it was fast. Jez Humble (00:19:33): Super fast and super iterative. And it was very rare that I would work on something that I wasn't deploying into prods the same day that I wrote it. That was pretty uncommon. And I went from that into this project where Thoughtworks was working on this thing, 70 people where we worked for months without anything getting deployed. And when we did deploy, it was over the weekend with a massive Gantt chart. And it's such a culture shock to me. And I was lucky to work on a team with a bunch of really smart and brilliant people, Dan North, Chris Read, Sam Newman and a bunch of other less known, but equally brilliant people, Dante Bryanes, Tim Harding, a couple of other people whose names escape me in the moment, but equally awesome people. (00:20:21): And yeah, that team came up with a whole bunch of ideas to basically make it suck a lot less. And that was the genesis of the Continuous Delivery book. And Dave Farley, my co-author, he was working on a different Thoughtworks project where, as you mentioned, the point of service also upgrades CD for point of service systems, equally hairy and miserable and difficult. And I think if you look at what Thoughtworks was doing at the time, we were taking extreme programming... Martin Fowler was our chief scientist. He's obviously the pioneer of refactoring, a big, big part of continuous integration, obviously a big part of XP in the early Agile movement. And we were taking those ideas and applying them in companies where you were not even allowed to say the word Agile, because it was too scary and you don't want to freak out the managers. (00:21:13): So it didn't feel like being part of a movement. It felt like being part of the Agile movement. That's how we identified ourselves, is just people who were implementing Agile in Enterprise situations. And I think what we found was, this part, the bill test deployment part, it was treated as an afterthought, but actually it's where a lot of the pain was. And so the book was just about how to solve those pain points. And then about two years into the writing of the book, Mike Nygard's Release It! book came out and I remember getting that book and then being, "Oh shit, he's written my book. He's beaten me to it." And I was genuinely, both upset, but also reading it. It's a great book. Gene Kim (00:22:03): It's a great book. Jez Humble (00:22:04): And I could have- Gene Kim (00:22:06): "What else is there left?" Jez Humble (00:22:08): But it turns out there was plenty lefters, and this is, I think... I often have to talk myself out of zero-sum thinking, and that was one of those occasions where I really had to do that. And you realize now there's room for so many books on this topic, but at the time, that was perhaps the first book that covered this topic and Continuous Delivery was the second book. (00:22:30): A year before we released Continuous Delivery, Tim Fitz's blog post on continuous deployment IMVU came out. I wasn't really aware of the velocity thing until... Actually, no, I was, I did find out about that because I asked John Osborne to review the book and write a praise quote for it. So all these things started coming together, but only two or three years into the book writing. And then 2010, when the book came out, we'd just had the first DevOps days, I guess. And it just so happened that we had. Jez Humble (00:23:03): ... you know. It just so happened that we launched it at the right place at the right time. But it was not through any planning, but it was definitely very exciting, when there was so much buzz around it and you realize, "Oh, there's a movement. This is actually a movement and it's an industry wide movement and it's transformational." We're seeing that play out, we've seen it play out, and we see it continue to play out now. But it wasn't through any planning. But it was very exciting when all these little things started popping out and it just turned into a wave. Gene Kim (00:23:34): It's funny. I was talking to Patrick and these have been so fun. I was asking him about that his project, where he had his aha moment. He was essentially the infrastructure person. His role in the project was essentially to disappear for two, three weeks racking, stacking servers, configuring things. So he said using slur zones, it was the first time that something that would have normally taken like a week, you can do in like 10 seconds. It was the first time that he could actually be a part of the team and it was transformative. I'd love hearing your story about this 2004 project. So when you talk about Chris Reed and Dan North, why you all? Was it because you were the leftovers, not good enough to be working on the real features? Or was it because gravitated to the problem? Why you all? Jez Humble (00:24:20): Full Works, UK, at the time had this position called infrastructure and environments. I can't remember what the exact title was, but like person basically. I got hired into that role. Julian Simpson interviewed me, Dave Farley interviewed me. So there's a whole bunch of people who basically realized that infrastructure management and engineering was a thing. I don't know who came up with that role. Probably some combination of the people I've just mentioned. (00:24:50): Dan North was a pretty big deal in the agile community in London at the time. There's a whole group of kind of people who were doing a lot of in London and they had regular meetups. There was Extreme Tuesdays, they would go down the pub and talk about eXtreme Programming. So there was already a community in London doing eXtreme Programming and talking about it. A whole bunch of people. So I just got hired into that role because they needed people who knew about infrastructure and software, and ideally agile, although I'd never really done agile by that point. And I just got thrown onto this project. (00:25:29): So it was completely random from my perspective, but definitely from Thought X perspective, they were intentional about this. And it came out of the fact that we are practicing XP, but we saw this gap, which was the infrastructure and environments piece. Gene Kim (00:25:44): Which warranted putting some extremely talented and experienced people on the team. Jez Humble (00:25:49): Right. They needed people who filled this kind of weird hole. And I filled that weird hole because I'd worked at a startup and I had to do everything. I've racked servers, I had to go down to the data center because I [inaudible 00:26:04] into a port and then brought down the interface to fix something and then disconnected myself and the rest of the internet from our production environment. So I'd done these dumb things and had to do coding and infrastructure all in the same day. And didn't really distinguish between any of this stuff, because it's just a problem that needs to be solved. It's just all technology. So I just happened to fit into that whole. Gene Kim (00:26:30): Awesome. Before we leave that topic, this is actually this question I've been dragging around for probably five plus years. We once talked about how interesting it was, how widely used PHP was. And that one of the interesting properties of PHP is that you can deploy one file at a time, very much like ASP.NET. Jez Humble (00:26:47): Yeah. [crosstalk 00:26:48]. Gene Kim (00:26:49): ... file, where you have to compile the whole thing and push out the whole thing. To what extent do you think that was an important property? And have we lost something by not deploy just one little piece of the property at a time? Jez Humble (00:27:03): Yeah. I actually believe that. I think that's a very insightful comment. I kind of joked that PHP is the original microservices architecture, because every file in PHP is effectively a microservice. That's kind of a joke, but there is some truth in it as well. And the fact that you can independently deploy the different pieces, I think is actually an interesting architectural property. (00:27:31): We researched this in the door research, of course, and we found that the ability to independently test and deploy individual components was one of several parts of that architecture construct, which we found strongly predicts the ability to practice continuous delivery. So we know that it's important from the research and it is an architectural property of that platform, which I think my experience backs up what the research says. Which is that, that capability is actually hugely important. (00:27:59): I think there's other things about PHP that are nice as well. There's no warm up time, there's no VM, it's all written in C. There's this huge library in PHP. It's so easy to get things done because there's this huge library. If you look at the code in the libraries, you'd probably throw up and be very anxious about your life choices. But the fact is that it is really powerful and it works. And the feedback cycle is so fast. I think that's the critical thing. (00:28:29): I'm sure you do remember Elizabeth Hendrickson giving that talk on the care and feeding of feedback cycles- Gene Kim (00:28:34): Yes. Jez Humble (00:28:35): ... at FloCon back in the day. I think that focus on feedback cycles, I think, was a changer for me. Because that ultimately is what it comes down to. And you just get these very fast feedbacks. The other famous talk, ah, I can't remember the guy who gave it. He was an Apple guy. He was talking about the ability to prototype and see how things change as you prototype it. This will come back- Gene Kim (00:28:58): You're talking about not Victor- Jez Humble (00:29:01): Yeah. Gene Kim (00:29:01): Bret Victor. Bret Victor, right. Jez Humble (00:29:03): There you go. He gave that talk about he built this little JavaScript game and he was changing parameters and the game was changing in real time. And his whole comment is like, if you can change something and see the change as you are changing it, it's incredibly powerful. And your ability to create things and build things that are good. I think PHP gives you that. Who knows to what extent it was intentional. Gene Kim (00:29:27): Gene, here. Super funny commentary on the common property, the PHP, ASP, ASP.NET and so forth. It's also so super interesting that all those properties are negated when you have to put them into a container where you are once again, having to upload a whole new container image, not being able to upload just one file at a time. Okay. Number two, I was so surprised that Jez brought up Bret Victor. I'm going to read from his Wikipedia entry. "Bret Victor worked as a human interface inventor at Apple from 2007, until 2011. He was a member of the small group of people who worked on the initial design for the iPad and contributed to the development of other products, including the Apple Watch. Victor received attention for his talk, Inventing on Principle, in 2012 and The Future of Programming 2013. A major motivation for Victor's work is to make it easier and faster to use complex tools and ideas." (00:30:23): I will put a link to his Inventing on Principle video, which is absolutely jaw dropping and shows a way to develop and interact with code, that just seems utterly magical. And I will also put a link to a page on worrydream.com that shows prototypes of his vision of how code should work, called Media for Thinking the Unthinkable. Incidentally, many people put Bret Victor's work into the same category as a seminal, Doug Engelbart 1968 Mother of all Demos presentation. I'll read from his Wikipedia entry. (00:30:56): "The Mother of all Demos is a name retroactively applied to a landmark computer demonstration given at the ACM IEEE computer society's Fall Joint Computer Conference. This 90 minute demonstration demonstrated for the first time, many of the fundamental elements of modern personal computing. Windows, hyper text, graphics, efficient navigation and command input, video conferencing, the computer mouse, word processing, dynamic file linking, version control and collaborative realtime editors. This demonstration was highly influential and spawned similar projects as Xerox PARC in the 1970s, the underlying concepts and technologies influenced both the Apple Macintosh and the Windows graphical user interface in the 1980s and 1990s." (00:31:38): Okay. So the point here is that there's such value you to be able to see what you're doing as you're doing the work, as opposed to being stuck in a batch mode where you write your code, compile the code, deploy the code, and then a couple weeks later you get to see it in production. Okay. Back to the interview. It's been six years since the first edition of The DevOps Handbook came out. What has been the most surprising thing for you since that book came out, six years ago? Jez Humble (00:32:07): I think the thing that has surprised me most is the extent to which Docker and Kubernetes have really taken off in a really big way and completely change the landscape. And also the extent to which I've been able to completely ignore them and still do my job. I still haven't used either of those technologies in [inaudible 00:32:31]. They've done stuff for me. I know that Docker is... containerization, I've used. Docker as a product, I've never used. Kubernetes, I've used that once. In fact, when I was working for the US federal government, we have this thing called the Federalist, which is a publishing platform and that runs on Kubernetes. So I I've kind of used it as a user, but I've never of touched either of those things really. That for me has kind of been interesting. Gene Kim (00:33:01): There's an irony here because you are now an SRE supporting Google Cloud Run, which is entirely dependent on containers. Jez Humble (00:33:08): Yes. Exactly right. But it's the concept of it, rather than the implementations. I think the concept actually, as you say, it goes back to [inaudible 00:33:18], goes back to Borg, obviously. So definitely the ideas have been very powerful, but those particular manifestations of those ideas, I've kind of been able to ignore. But it has been really interesting to watch that whole thing play out. Containerization, Docker, Kubernetes, service meshes. I remember working in financial services in London, in the early noughties, compute grids were the next big thing. And that never really happened, but now we have Kubernetes and people are doing the same kind of ideas. Gene Kim (00:33:56): In fact, it's funny. I remember, I think all years, at least from 2013 to 2019, the State of DevOps Research surveys were done on PHP in a, was it Linode? Jez Humble (00:34:08): I think it was actually Gandi, I used. It's a French hosting company Gene Kim (00:34:13): In the first year, I remember you used this 64 K memory partition. Jez Humble (00:34:20): You made fun of me for doing that, which was entirely merited. But it's just who I am, I can't help it. And my wife's always criticizing me for this, quite rightly. Which is I'm very parsimonious about dumb things. The things that I care about being parsimonious about, are the wrong things to be parsimonious about. But I really care about them. So I'm constantly trying to deprogram myself from that. But that was a great example of that. Gene Kim (00:34:49): If I remember correctly, like in 2015, I think you did acquiesce and increased the size of the VM to... I don't think you refused to go any higher than half a gig. Jez Humble (00:35:00): Yeah, I think that was my limit. I thought that was being extremely... I was really over engineering that thing with half a gig. Gene Kim (00:35:07): So in The DevOps Handbook, we have a section that's really about principles and we a set on patterns. What has been your favorite pattern that went into The DevOps Handbook? Jez Humble (00:35:19): One of the things that we talked about in the second edition that we didn't talk about in the first edition, which I'm a huge fan of, is some of the work that we did in 2019, where we looked at adoption basically, and transformations and patterns of transformation and how companies adopt the ideas that we talk about in the book and how we looked at dojo's and communities of practice and prototypes and all these other training institutes and centers of excellence and all these things that people use to implement transformations of all kinds, not just DevOps. That was pretty interesting. So yeah, in 2019 we looked at transformation strategies, training center, center of excellence, proof of concepts, communities of practice, big bang, mashups. And what we find is that the elite for performs tend to use community structures. They basically build community structures in order to help adopt ideas. That's things like proof of concepts as a seed, communities of practice, grassroot stuff. So they're building community structures to share knowledge throughout organizations. And this is probably going to tie into some of the work that you are doing, I imagine, with Dr. Steven [inaudible 00:36:40]. (00:36:41): But it was really nice to see that, that community structures are actually really important in driving transformation. We tend to think about transformations being either a bottoms up or a top down thing. And this shows that it's actually more complex than that. Also, not to say that things like sense of excellence and [inaudible 00:37:02] can't work. We see those things in elite performers, but we see them as part of that kind of community structure pattern. So that I think is really interesting. Gene Kim (00:37:14): That's interesting. And so if we were to sort of think about the organization as a network, these are structures that kind of connect the edges outside of kind of a traditional hierarchy. Is that what's in your head. Jez Humble (00:37:25): Yeah, I think there's two things. Firstly, it's connecting different parts of the organization that might not be normally connected. So communities of practice is an example of that. It's where you get people who might not normally talk to each other to get together and share their stories. But also, it connects different parts of the hierarchy. You're not just connecting laterally between different groups, you're also connecting up and down within the organization as well. Because leadership has a really important role in nurturing and developing communities of practice and hopefully drawing from them and helping push those lessons elsewhere. It's connecting all kinds of different parts of the structures I think. And that for me is what's cool. Gene Kim (00:38:09): In fact, our good friend, mutual friend, Elizabeth Henderson, she said, "You can measure the intelligence of an organization based on the number of connections and the frequency and the intensity of those communications." These are the structures that create more of those connections and meaningful interactions between them. Jez Humble (00:38:27): That's great. I love that. Gene Kim (00:38:28): So one of my favorite parts of The DevOps Handbook, is just how many case studies we have. We have a whole bunch of came from the tech giants, Facebook, Google, Amazon, but we had so many from large complex organizations. And we had 18F in there as well, about the compliance journey that you were on. So that's certainly one of my favorites. How about you? What is one of your favorite case studies in The DevOps Handbook? (00:38:51): Gene, here. I forgot to define 18F. 18F is a technology and design consultancy for the US government, inside the US government. 18F partners with agencies to improve the user experience of government services, by helping them build and buy technology. I'll put a link to it in the show notes. Okay, Back to Jez. Jez Humble (00:39:09): So obviously I'm very attached to the 18F case study that we put together. It was a huge privilege and a ton of fun to be working on the cloud.gov team that put together the platform-as-a-service that we built that was used by many different federal agencies, actually. And to build a platform as a service that was based on one technology that we used continuous delivery in order to develop and deploy and to show that these ideas work even in a highly regulated context. (00:39:37): So definitely I have a soft spot for that and for the team that I worked on and for doing that work and the impact that it has still around today. If you are a federal agency, you can still go to cloud.gov and sign up and use it. And I think there was some work ongoing to make it available to state and local governments as well. I don't know how far along that is, but you should check out if you'll listen to this and you work for a state or local government and see if it's available. (00:40:05): I think in the new edition, the case study, early, we talked about community structures. There's a case study that we put in the new book, where we talk about how he did that at Google. This has been talked about by Mike Blands in a blog post on Martin Fowler's blog, where he talks about heart bleeds, a very long article that starts talking about heart bleed, but actually kind of breaks off in the middle to talk about how testing culture was developed at Google and the testing on the toilet program. Basically, how they created a community of practice around testing at Google. Developer led testing. And that testing on the toilet program, where they basically printed a newsletter off and stuck it to the back of every toilet cubicle to kind of educate people about testing, so you could never escape testing. And how basically- Gene Kim (00:40:55): And here's who to call to learn more. Jez Humble (00:40:58): Yeah, exactly. That was hugely impactful. And actually, there was a paper that came out recently, which looked at testing on the toilet. We can maybe link to that in the show notes. Gene Kim (00:41:08): Absolutely. Jez Humble (00:41:10): I think that's interesting as an example of communities of practice. But also as an example of how even companies like Google, that everyone kind of imagines are kind of perfectly formed and sprung from the head of the use, practicing all these technologies. Google, I'm not saying anything that's not in the public domain, Google did not have a culture of comprehensive testing and test automation and it developed one and it didn't develop it by the management telling everyone to implement it. They developed it because Google gave people 20% time. And a bunch of people were really passionate about this and they didn't have resources, but they had passion and they kind of tried all these kind of unusual techniques to create this culture of testing, which is very strong today at Google. But that's something that anyone can do. It's not something that Google does by din of being Google. It's something that Google does indirectly by giving people the resources and the time to try things out and experiment. Gene Kim (00:42:10): You're saying, if you have printers in your organization and toilet stalls, you too can pull off your own testing on the toilet program. Jez Humble (00:42:18): That's right. Harder to do in a kind virtual, remote first setting. Gene Kim (00:42:25): And I will put a conspicuous note that Mike Blands' book about this at the 2015 DevOps Enterprise Summit. Jez Humble (00:42:31): Yeah. Gene Kim (00:42:32): Awesome. Jez, I have to tell you it's been so much fun. My cheeks hurt from smiling so much. Is there anything else that you want to share? Jez Humble (00:42:46): As always, it's been an absolute pleasure. We haven't caught up so much the last year or so I think, which reflects just the level of craziness I think that that has been going on. But yeah, it's been great to work with you again on DevOps Handbook's second edition. Really happy with how that's come out. Very proud to be a part of the evolution of that. Really happy that we got Nicole's contributions- Gene Kim (00:43:07): Nicole. Jez Humble (00:43:09): ... and that she's played such a huge part in the second edition. That's great. I think that's going to add a lot of extra depth to the content. I'm very pleased with how that's come out. Yeah, no doubt this is not the end, but just the beginning. Gene Kim (00:43:25): It was so great talking to Jez just now. After a short break, I will be speaking with the last of my co-authors, Dr. Nicole Forsgren. Here at IT Revolution, we've been hard at work bringing you in 2021 two DevOps Enterprise Virtual Summits, this podcast, two issues of The DevOps Enterprise Journal and a new immersion course from Dominica DeGrandis, renowned author on flow efficiency, who dives into the fundamentals needed to help you better understand your organizational workflows and make them more efficient. (00:44:04): And we just published the second edition of The DevOps Handbook, which this episode and the last episode are all about. I mentioned that as much as I love the first edition of the book, I love the second edition, even more. The 15 new case studies, mostly from the DevOps enterprise community, including Fannie Mae, Adidas, American Airlines, and the US Air Force. There's over 100 pay pages of new or updated content, including so many of the solidified learnings from the State of DevOps Research and the Accelerate book. There's a new forward and materials from Dr. Nicole Forsgren. There's an updated afterward from all five co-authors and there's a new section with new resources at the end of each part of the book. (00:44:47): I want to thank Dr. Nicole Forsgren who joined the authors team. On top of everything I mentioned above this expanded edition includes new material from her. She's a good friend and a renowned researcher. She is currently partner and VP of research and strategy at Microsoft and was lead researcher on the State of DevOps Research and lead author of the Shingle award-winning book, Accelerate. In this world we're living in where we need to adapt more quickly than ever and create resilient organizations that can respond to turbulent times, in order to help our organization survive and win in the marketplace, the topics covered in The DevOps Handbook, second edition, are more important than ever. (00:45:24): Okay, let's get back to the interview. I met Dr. Nicole Forsgren in 2013, right as we were in the planning stages of the second State of DevOps Research study, which was continuing the work that Jez Humble and I had started with Puppet the year prior. So this is the work that eventually went into the 2014 State of DevOps report, which was full of so many amazing breakthroughs. It included the famous Western Organizational Typology model, showing that culture was one of the top predictors of performance. And you'll hear the story of how she created this instrument in this interview. (00:45:59): It was also the year that introduced the organizational performance model into the- Gene Kim (00:46:03): And it was also the year that introduced the organizational performance model into the research where for the first time we were able to test a hypothesis that technology performance could be linked to organizational performance as well. In other words, the finding was that high performers not only had these amazing technical measures, so that's deployment frequency, deployment lead time, changes accessory and meantime repair, but those organizations were also twice as likely to achieve organizational and mission goals as measured by meeting or exceeding profitability, market share, and productivity goals. This was such an important finding for us because we wanted to show that DevOps creates business value, that we were solving a business problem, not just a technical problem that could be safely delegated away. And this was a finding that actually got us mentioned in The Wall Street Journal, which was such a huge moment for us. (00:46:50): I've described many times that the seven years that we got to work together on the state of DevOps research is one of the achievements that I'm most professionally proud of and where that study went simply would not have been possible without Dr. Forsgren. She is a world class researcher. And throughout the years, she was on the project. She was the principal investigator. She is not just an expert in technology and DevOps, but she is also expert on research and survey methodology, survey design, analysis, and so much more. So I'm looking at a presentation that Dr. Forsgren, Jez and I did together at the Velocity Conference in June 2014. And one of my favorite slides described what some of our goals were. It was to do industry and academic research that went beyond just anecdote, peer recommendation, description of prior experience, return on the investment stories, description of best practices or benchmarks. (00:47:43): Instead, it was to do population studies and academic studies. So over the years we used the population study or cross population instrument, which is the same type of instrument that the medical community used to determine the link between smoking and early morbidity. Dr. Forsgren made that possible by bringing her incredible sense of rigor and deep expertise. As I alluded to in my previous interview with Jez, this period of time was an incredibly productive one. And I have such fond memories of those frenzied working sessions as we craft these hypotheses, design the survey instrument, watch the responses come in. And then once we had the complete population, we would start the analysis and see whether these hypotheses were confirmed or not. Throughout this process, we would have vigorous argumentation about whether we could cut the data one way or another, arguing about the wording of the findings and so much more. (00:48:35): And without a doubt, the research benefited from all that attention to detail and rigor. Dr. Forsgren was such an advocate of the research goals, but she also played the role of referee, holding all of us to the same standards that an academic journal would apply. And you'll hear more about many of these aspects of the work in this interview. So I hope I've conveyed to you that Dr. Forsgren is so good at so many things, but I do want to share one memory that I have of these times, which is the utter awe and amazement I have every time I watch Dr. Forsgren use SPSS and Microsoft Excel. So I've been using SPSS since 2005, and I feel like I've done some pretty cool things with it, including writing awful basis code to beat data into submission. But to see Dr. Forsgren use SPSS for real analysis is like watching a world class pianist perform. It is utterly jaw dropping. (00:49:32): So Dr. Forsgren is the lead author of the Shingo Research Publication award-winning book, Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. She also led the commercialization of this work through the DevOps Research and Assessment Organization, affectionately known as DORA. DORA was purchased by Google Cloud in 2018. And she is now partner and VP of research and strategy at Microsoft in the famous MSR Microsoft Software Research Lab. And I am so excited that she was willing to lead the effort to create a second edition of The DevOps Handbook. So without a doubt, without her leadership, I'm not sure if a second edition ever would've been created. So I am so excited that she was willing to lead the effort to create a second edition of The DevOps Handbook, for reasons [crosstalk 00:50:25]. Dr. Nicole Forsgren (00:50:25): For reasons that evolve. We are wonderful collaborators. I am incredibly grateful. Gene Kim (00:50:31): Oh, and right back at you. So I've introduced you in my words. So can you introduce yourself and describe what you are working on these days? Dr. Nicole Forsgren (00:50:38): Sure. I run a research lab, Microsoft Research. It's called Developer Velocity Lab. And what we do is we investigate things surrounding developer productivity, community wellbeing, and some of the really exciting things that are happening around AI assisted development. Right. So like when we talk about productivity, there's always this really interesting thing that happens around productivity's hard to measure or people are like, oh, well that's easy, right? Like lines of code, which you and I know is a nonsense metric, but sure. Right. Roll with me. Lines of code. Sure. Okay. What happens when you've got AI assisted development? Whether it's assisting with things like co-pilot is now a thing or many other things like, VS Code and IntelliSense, or we have things like tools that help us with merge conflicts. So sure, let's roll with lines of code. (00:51:32): Well, what happens when lines of code is entirely out the window, right? So we develop the space framework earlier that help us come up with more holistic measures or the good day project, right, when we're not talking about teams and organizations, but how do we help people have good days and good days more reliably and more consistently. So that is what I'm working on now with an amazing team. And it's also a cross organizational team. So it's super exciting. But before that, as you know I was working with DORA and we were looking at kind of the ship and ship to production process. And before that, some of my foundation was as a developer, I was a software engineer, [inaudible 00:52:12] admin. I even worked on mainframes. So I kind of have been around. I was a professor even so. Gene Kim (00:52:17): That was awesome. By the way, you had mentioned the famous and storied Microsoft software lab. What has it been like to work within that organization? I mean, it's been the source of so many famous publications and things that advanced the profession. Dr. Nicole Forsgren (00:52:33): Yeah. I have to say, so being at MSR, Microsoft Research has been incredible. It's wonderful because you're working with some brilliant people, people who are just foundational to their entire fields. I'm working with MacArthur geniuses, who I've got a quick question and they'll just open up the calendar and we just sit and nerd out. And I'm asking them questions and they're just so delightful and delighted to talk to me. And I'm just like, my mind is blown and my whole day is made. (00:53:12): And people who like I said, they've been foundational to their field and they're so kind, and they're so lovely. And it's almost just this mind melting experience because I've also been in academia where maybe I'm jaded. I'm totally jaded. Because I've also worked with people who are foundational to their fields and they're jerk faces. I don't want to be here. But at MSR people are collaborative, and open, and kind, and finding ways to help you. And it's this incredible experience where truly more people together make and create better things. And it's for the better of society and for the better of you and I am so grateful every day. Gene Kim (00:53:59): In fact, this reminds me, I remember my first time when I got to visit the software engineering Institute at Carnegie Mellon University, this must've been 2004. And I left, I spent three days there and I left in this euphoric state. Everyone is so much just, reminds me of what you described about. It sounds like people there are happy. It's a generative happy organization. Dr. Nicole Forsgren (00:54:26): Yeah. It's really wonderful. Gene Kim (00:54:29): Oh, that's awesome. In fact, I still have my Software Engineering Institute mug that just said, because I love the work. Dr. Nicole Forsgren (00:54:39): Yeah. And to your point about generative, right, like I've even had calls with people who say, so you can't do it this way. Like that's really not going to work, or that's not going to fly, or this cannot be a thing. Right. Even going through IRB, our internal IRB. And let me help you find a way to get there. So it's this incredibly collaborative building environment where people are offering critical, critical feedback and helping you find a solution. But it's also not this weird thing where it's like, don't come with problems, come with solutions. No, sometimes you got to come with a problem and say, this is going to break your project or you can't do this. Okay. Now let's brainstorm together or go ahead and brainstorm off. I got to run. Okay. Now let me help you find this thing. And I'm like, my work is getting so much better. Or if I point out a flaw, no one's mad. People are not mad. Gene Kim (00:55:32): That's awesome. By the way, what's an IRB? Dr. Nicole Forsgren (00:55:35): Oh, Institutional Review Board. So anytime we're dealing with data or systems that involve humans in any way, how do we make sure that we review it so that it's protecting humans, it's ethical to humans. We're thinking of any possible way to make sure that we make our systems better. Gene Kim (00:55:52): Awesome. Awesome. So great. Congratulations by the way. You deserve it. Dr. Nicole Forsgren (00:55:57): Thank you. Gene Kim (00:55:58): So can you tell me what it's felt like for the state of DevOps research findings to be seemingly at the tip of the tongue of most technology leaders these days? I mean, holy cow, it was actually in the S-1 filing for GitLab as they were filing to go public. Tell us about that. Dr. Nicole Forsgren (00:56:15): It's been quite a ride, right? It's been this amazing journey where when we started this and even when I think back being in my PhD program and trying to think about what I wanted to do and taking this huge pivot into this, I really just wanted to find a way to make developers' lives better. And then how could I extend that and think of the various ways to do that, right? Because we need to make their work better. We need to make their daily lives better. We need to make their jobs better. We need to sell it to organizations. And then as an academic, we need to be making an impact, for some definition of the word impact. Right. And when I was in academia, they were used using the words, broad measures of impact, right? Because it only used to count if you could get citations, get a publication and get citations. (00:57:04): And so now when I think about this cute little baby research project, and I hate to say that, shout out to the puppet team who first thought of this and then Alana, God bless Alana because I rocked up and I'm like, no, let's do this a different way. And like deconstruct all of these measures and introduce all this academic rigor. And I was like, trust me if we do it this way, it's going to be, we'll get a twofer. And then Alana still did her like marketing and PR and design magic. And we brought this incredible team together. And now it has turned into something that when I think about broad measures of impact, this is a hell of a broad measure of impact. Right. And when we think about what we've been able to do as this like small but mighty team and how it really has kind of changed the industry. Executives use it, tooling teams use it. It's turned into this industry standard of how do we think about and measure and gauge systems, right? (00:58:00): It's hidden S1 filing, open source systems use it, for profit systems use it, surveys use it, telemetry systems use it, and peer review papers use it. And I'm so grateful that we could start with something and rigorously measure it year over year and find such great signal. And some days I sort of pinch myself. And at the same time, I'm like, Hey, what else can we find that's better? Can we find something that's better to replace it? And then also it's this thing where I'm like, we haven't found something yet. There's some really great signal here. Gene Kim (00:58:38): Awesome. And how has it felt? Dr. Nicole Forsgren (00:58:41): I think, I don't want to say validating but rewarding. I think even rewarding. Gene Kim (00:58:47): Yeah, totally. Dr. Nicole Forsgren (00:58:48): I mean, we used the measures in this year's Octoverse right? So we surveyed in the data that kind of coalesced over 12,000 people across core GitHub users and open source and it still held. So like so many different users and across so many different systems and it's such a solid signal. It's such a good signal. Gene Kim (00:59:11): Awesome. Awesome. Ah, the mark of great theory. So we've been at this for nine years now. So what for you has been the most surprising, delightful, insert your adjective of choice, finding since the state of DevOps research studies have come out? Dr. Nicole Forsgren (00:59:29): So I'll say this and you may react, I would say organizational performance, which is almost funny now. I think we take it for granted, but when we first tossed this measure in, when we were first working together, I don't know if you remember, but I was like, I'm going to try this. I think, back then we were calling it, IT perf now we call it software delivery performance, but these speed and stability measures, I think it's going to impact the way organization perform. And I just threw it in. I grabbed this measure out of the accounting literature because that's how we measure the way organizations perform, is on their financial statements. So I threw it in there as like a, maybe a throwaway measure. I included it. And I was like, it's probably going to fall out. (01:00:10): I don't think it's going to be statistically predictive. I don't think it's going to hold, but I think there's something there, right? Because we've been around this DevOpsy thing for a while. 40 years of academic literature says this isn't going to work, 40 years. And it did. And the reason that it did is because DevOps is fundamentally different, right? It's not just ROI on figure quoting here, right? It's not just ROI because you don't invest some money and walk away. It's because it's a fundamentally different way of doing work because you have to have the tech, you have to have the process, you have to have the culture. It has to be this holistic core transformation where you're creating a sustained competitive advantage. (01:00:57): And it was such a surprise. And I think that's really one of the reasons we hit so much press. We had The Wall Street Journal. And now, like you said, we're nine years in and people are like, well, of course it makes a difference. Of course, you see two extra turns. Of course this is a thing. But that was also really where I was like, oh, this is it. Because not only do we reduce, I think that first year we maybe did like deploy paint or something. We had early, early signals that it was actually making lives better. Gene Kim (01:01:31): So I'm not sure if you remember this, but I was a believer the whole time, maybe even naively. But I remember showing you the book by Paul Strassmann, so who studied technology for going on 30 years. He was a CIO of the Department of Defense, of Kraft, of Xerox. And he spent 20 years of his life looking for the return of investment on tech technology. He was doing things like counting paper that are coming off of copy machines. He's looking at technology spend and he could not find it. And yet for over 10 years, the reason we had done the benchmarking work before was this quest to try to find the signal that technology could improve business performance. (01:02:09): And so I remember seeing you put that instrument into the survey and just how you crafted the hypothesis. I mean, it was amazing. So I remember, I think that was like one of the findings where we were on a, what were you, it was before Zoom, you were sharing your screen in SPSS as we were generating the findings together. Just that feeling of seeing that signal. I mean, it was just elation, euphoric. I mean, just one of those, I felt like it was the quest that was not coming to an end, right, but showing that the quest was real. Right. Dr. Nicole Forsgren (01:02:49): It was there. Yeah. Now, I will say Erik Brynjolfsson at MIT had been doing some early work. So he was one of the first to find this signal too. But there really had not been much. Right. We've got like 30, 40 years of like, we know it's there. We're not sure. We can't quite capture it. Erik Brynjolfsson had found one or two studies where there was some there where we know it's there, but how do you capture it right? And I think that missing link was software delivery performance. Because you can't go from investments in IT, right? Because you don't do continuous delivery and make money. You do continuous delivery so that you can do things better, whether it's better features, recover faster. Right. It's what you deliver to your customers in order to make money. It's what you do for your users in order to make money. So it's that missing link. Gene Kim (01:03:37): Ah, so good. Dr. Nicole Forsgren (01:03:38): Which is the DevOps. Right. Gene Kim (01:03:42): So awesome. In fact, I'm having this big grin as we're talking about those set of finding associated with that. By the way, while I have you on the call, one of my favorite ones is architecture. And I just remember the words, architecture was one of the strongest predictors of continuous delivery, even more so than automated testing or version control, which I just loved. And by the way, I think I sometimes speak and say is one of the top predictors of performance. To what degree can you assert the importance of architecture? Dr. Nicole Forsgren (01:04:13): I mean, I think architecture is super important, right? Because it ends up being entangled in so many of those other things, right? Architecture enables so many things. It's really hard to do continuous delivery if you have a really tough architecture. Now, the other interesting thing though, is that architecture, if you have a well architected system, it enables many other things. So you can do many things on a mainframe if you think about your architecture well, right? Because if you have, even with a mainframe, if you think about your architecture and you find ways to have things be loosely coupled, which I say, so I was on mainframes. If you have a lot of sub routines, you can start treating that mainframe like sort of a loosely coupled architecture system. Gene Kim (01:04:59): So good. So we talked about the most important and most delightful finding of organizational performance. So those performance measures are caused by patterns. Like I said, we divide them up into technical practices, cultural norms, and architecture. So what is your favorite pattern? The things that actually influence performance. Dr. Nicole Forsgren (01:05:16): I think there are a few, but it may come down to culture for me because for a few reasons, one, because it's hard another, because culture can end up playing a role so many different places and you can't just throw money at it. Right. So, and because it influences so many things. Right. Because we always have the insert Conway's law joke, DevOps bingo. Right. But you can make changes throughout the culture. You can make changes throughout the organization and it ends up imprinting itself so many different places, right? It's almost like shadow IT. You've got shadow culture, right? (01:05:56): So many times people think they can just throw money at tech and it will be fine. If you throw money at tech and you have a broken culture, you can only get so far. It's just not going to work. And if you can find ways to fundamentally change your culture, you can get really, really far with whatever tech stack you have. Right. I've seen people do magical things on mainframes. Right. I was meeting with the Navy a couple months ago and they're doing incredible things with very old tech stacks that they cannot change because they are embedded in ships. Right. Gene Kim (01:06:35): Yeah. Dr. Nicole Forsgren (01:06:35): You can be fundamentally innovative and you can find ways to make things work when you've got that culture figured out. People are just, the ingenuity there is incredible. Gene Kim (01:06:48): And what did you learn about culture in this journey? Dr. Nicole Forsgren (01:06:53): So the research has done a lot for me. Right. We've found a few things that are predictive. There's definitely kind of a reinforcement loop there. I learned a lot by meeting with a lot of teams, development teams, executives. I learned a lot by just meeting with so many teams in so many different industries, highly regulated industries, startups, top secret fields, culture is going to make or break you. And if you let people learn, and grow, and find very creative solutions, you can just make magic happen. Right. I've seen people do the most innovative things in air gaped environments. Right. Like over the year updates in air gaped environments, and like magic is happening or. Gene Kim (01:07:42): Where you literally cannot see the other side because your not allowed to. Dr. Nicole Forsgren (01:07:45): You literally cannot see anything happening. And they're finding incredibly creative ways to make things work. Some of the things that I found are that when you give people space, you need space, you need permission, you need empowerment to find solutions. I loved that part of The Idealcast with Dr. Westrum. It's right. Right. I mean, a bad leader can just poison a well in ways that are really difficult to recover from. Sometimes you can, like if you can just kind of umbrella or shield an organization that can sometimes work, but it's from what I've been able to witness, it's almost amazing how quickly a toxic leader can just poison an entire environment because fear can kill almost anything. And then there's no incentive to do good things. It's just too hard. Gene Kim (01:08:50): And by the way, I have to thank you for that introduction to Dr. Ron Westrum. In fact, I told the story before, but I remember that feeling when I saw that email where you had CC'd me on a thread. Gene Kim (01:09:03): Email... Where you had CC'd me on a thread where, that involved Dr. Ron Westrum. And I remember texting you saying, "Is this who I think it is?" See [inaudible 01:09:11] I reach out? I mean, can you tell us a story behind that? Dr. Nicole Forsgren (01:09:14): I am so grateful to him. He is so wonderful. I have to say I was this adorable little baby junior faculty member, and I shot him an email out of the blue and he is this, emeritus, genius, senior faculty. And I was like, "By the way, I'm using your work in some of my research. Thanks so much. Here's what I've done. Here's the paper." And he emailed me back. I was like, what are you doing? He was just kind and lovely. And then we kept in touch over the years, every two to three years. He'd email me and say, hi. Asked me if I had anything to share, asked me for results on validity and reliability checks for his work. Because we had turned his typology, which was a table, into valid, reliable survey questions and measures. (01:10:09): I was like, "By the way, this is now industry standard. By the way, here's a tweet." Someone's like, "It's now known as the Westrum typology." Because listen, I'm a researcher. I'm not creative. I don't name things. So we just called it the Westrum cultural model. No one wants to use anything else, by the way, you're super famous now. I mean he was before and he was just lovely. And then I think you wanted to meet him. So I just shot you an email in an intro. And he got right back to us. Not that he's not super busy because of course he is. But he was so kind. Gene Kim (01:10:41): By the way, those four hours of interviews. To me, it was just so dazzling and it... I find so gratifying that he's now showing up in The DevOps circles as well. He spoke at the [LaunchDarkly 01:10:51] conference. So I'd love that people now see him, not just as a table, but as the person behind the table, which is just so amazing. Dr. Nicole Forsgren (01:11:01): He is so lovely. Gene Kim (01:11:04): Agreed. Awesome. What do you feel is the most important thing that you learned in this journey that's about culture, but what's been the most delightful learning for you? Dr. Nicole Forsgren (01:11:15): Yeah. I think a few things. One is that you can really give people tools and you can influence a broader set of folks and you can empower them and you can make a big change. So for that I think about [inaudible 01:11:34] report. This broad set of work that we had was at its core, a research project, right. And if we find ways to make it accessible, easy to read, easy to use, you can help people do great work and move pretty quickly. I did it as an academic work, but when we found ways to make it almost like an adult picture book, big font, big tools, work with a copy editor so that if people want, they can screenshot or take a single page. We didn't have line breaks in work places. (01:12:14): You can easily drop it into a slide deck or a board deck or anything else, use pictures that are easy to use. We can help and we can empower anyone to use this really easily. So I thought of this strategically, we tried to write it for three audiences, developers and [eng. 01:12:33] managers and executives. So we always tried to have something that a developer could use. So what makes CI impactful and predictive and useful? Every check-in of code has automated tests, automated builds and is green. Any dev can do that using any tool, any vendor can build something on that. Eng. Managers can do the same thing. So it's like you had actionable guidance, executives. What is org performance? if I'm or cloud, if I'm making investments in cloud, what are the five things I need for any cloud? (01:13:06): And why do I care? Because you're 23 times more likely to be an elite performer than not if you're doing cloud right, right? So that was nice. And then when we commercialized it, when we created [Dora 01:14:44], benchmarks are there benchmarks are candy, everyone's a benchmark. Everyone wants to compare. But when we had that priority matrix, we basically tricked people into strategy because everyone wanted to talk to us to say, "Well, what should I do next?" But that's consulting and consulting didn't scale. Now, God bless our partners. We had a wonderful partner network to help build out detailed roadmaps. But what could scale was, what should I do next? How should I think about prioritization? Okay. Well I could give you three to five to seven top things to think about instead of 50. And then I could have a quick discussion with you for 30 minutes or an hour to help you think through what was most important in your context. (01:14:02): And so giving... And then once they were done with us, they could run it at scale internal to their organization. It was a SAS based tool. And so I think realizing and learning that for myself was finding ways to give people research based tools that they could use and then they could run and they didn't need us anymore. Right now all of this is available online. It's basic, it's free, right? The quick checks online, all of the Dora base tools are online. People can make themselves better, really quickly. Gene Kim (01:14:36): Based on science. Dr. Nicole Forsgren (01:14:37): Based on science. And stories are great because stories and numbers together are great, but anyone can get better. Gene Kim (01:14:44): Awesome. I had mentioned that I'm so delighted that you were able to lead the effort to come up with a second edition of The DevOps Handbook. And by the way, I love the book. It's just, I really love the book. Dr. Nicole Forsgren (01:14:59): It's so fantastic. Gene Kim (01:15:00): And it's so much better now. And I think one of the reasons why I love the book so much is the fact that it has so many case studies and there's even more case studies now in the second edition. So what is your favorite case study in The DevOps Handbook? Dr. Nicole Forsgren (01:15:13): I think my favorite case study is the one that shows the power of using these methods that we call DevOps. And it's the one from the hospital. Gene Kim (01:15:27): Dr. Chris Strear. But yes, Dr. Chris Strear. Dr. Nicole Forsgren (01:15:31): Dr. Chris Strear, so we were able to chat and he highlighted the power of so many of these approaches because it applies anywhere. So many times people are like, "Oh, well this won't work here. Well, this won't work for me. Well, this won't work for my team." Listen, if these approaches work in a hospital, it's definitely going to work for your team because it's about thinking about what tooling you have and how you can apply this to the tooling at your disposal. Okay. So maybe the tooling that we talk about in this chapter, isn't exactly what you have. (01:16:11): Okay. Well, how do the principles that we talk about apply to the tooling that you have? How do the processes that we talk about apply to the process that you're using. And when we talk about the culture, how does apply, how does it apply most closely to the culture where you are? And that's the part that I really loved because anyone can make improvements and it's usually not a small improvement. It's usually a huge improvement. And that's the one I love because it can just blow the roof off of anyone's thinking. Gene Kim (01:16:45): That's amazing. I mean, just by reducing the number of ambulance diverts, the increase in employee engagement ability to retain a talent. I mean, it's just such a great story. Dr. Nicole Forsgren (01:16:55): I also have a friend who is an orthopedic surgeon and he read Accelerate. And I was like, "What are you doing? Why are you reading Accelerate?" And he was like, "A couple of the really technical chapters, I didn't really quite get." But he said many of the other things really made sense to him and really applied. And I was like, right, again, same thing. Gene Kim (01:17:16): Absolutely. And by the way, one of the things I love about that Dr. Chris Strear, a case study, actually it was in the talk was that the end of the initiative came with the change in leadership where these things were not valued. The sad part of the story is that after he moved on, this amazing story he tells about this incredible breakthrough performance over three years vanished. And they [inaudible 01:17:40] back to the not great states that where the story began and it just shows the importance of leadership. Dr. Nicole Forsgren (01:17:48): So much of it is about leaders. And the other thing I really loved about that story was when it was going, it was going really well. It went from it being so difficult to get nurses, to nurses wanting to work there. It says so much about work culture. And that's an incredibly difficult field, same thing with development right now. Look where we are and everyone's burning out and it's such a difficult time to work. It says a lot. And I love that part. I always want to look for the human element because as Dr. Christina Maslach always says, "When you've got canaries, don't replace the canaries, fix the mine." Your people tell you a lot about your work environment. Gene Kim (01:18:36): Well, as a side, I was talking to the chief operating officer of Legacy Health here in Oregon. And he said, they're actually... You take this very tight labor market in nursing already. Then you introduce mandatory vaccination. And a surprisingly, a high number of people are quitting. They're quitting because of mandatory vaccinations. I saw a statistic that was like one out of five people in healthcare have quit. So just to quantify exactly what you're talking about, dynamics in the healthcare industry around labor shortages, it's just mind blowing. (01:19:13): Okay. Gene here, Dr. Forsgren mentioned the amazing Dr. Christina Maslach whose research both she and I love. Dr. Maslach is a Professor of Psychology, Professor Emerita and researcher at the healthy workplaces center at the University of California, Berkeley. She's the author of the amazing book, the truth about burnout and has developed a leading research measure called the Maslach Burnout Inventory. (01:19:38): She gave an amazing talk at the DevOps Enterprise Summit, London 2018, where she talked about her lifetime of work in this domain. She talked about how burnout is a hot topic in today's workplace, given its high cost for both employees and organizations. And she talked about her research and the empirical findings that show that burnout is largely a function of the social environment in which people work. And there are six critical areas of mismatch between the person and the job. One of the most amazing lines in her talk is when she talked about how the majority of the literature and discussion in the workplace is on the Canary suffering burnout about how we can build more resilient canaries when the real discussion should be about the coal mine, which is killing all the canaries. So I will put a link to that video in the show notes, I'm also going to put a link to another amazing panel that happened the following year with Dr. Maslach, Dr. Forsgren and Dr. Andre Martin, who at the time was the VP of people dev at Google. (01:20:33): There is some fantastic discussion of burnout in this panel, as well as about workplace engagement, where I got some incredible insights on the ideas that are behind the second ideal of focus, flow and joy, which appear in the unicorn project. Number two, Dr. Forsgren mentioned the U.S. Navy case study about work that involved, deploying into top secret environments. This is such an interesting use case because apparently it's very common where the people developing the application don't have the clearance to actually use, or even see it running in the top secret environment. For this reason, I was so excited in 2018 when [Dan Gaffer 01:21:09], former DISA PM for the forge.mil Program and Jeff Payne, CEO of Coveros described how they were able to do exactly this for the forge.mil program for the defense information systems agency, which among other things manage many shared services that span the U.S. Department of Defense. Forge.mil is the Department of Defense's collaborative development platform for rapid delivery of dependable software services and systems in support of net centric operations. (01:21:37): Most striking about this experience report is how they managed to build a CICD pipeline that deployed into a production environment that they couldn't ever see. Number three, Dr. Forsgren also mentioned the case study by Dr. Chris Strear. So he's a medical doctor, not a PhD who I mentioned in my Idealcast episode with Trent Green, chief operating officer of Legacy Health on how the mass vaccination clinic in Portland, Oregon was able to increase the number of vaccinations from 100 an hour to 1300 an hour. So Dr. Strear was with me as we toured the mass vaccination clinic. He is now a chief medical officer at Columbia Memorial, and he gave a phenomenal presentation at the DevOps Enterprise Summit 2021 on the work that he did to improve flow within an emergency department. I'm going to read a couple of excerpts from that case study. (01:22:29): He described how the conditions in the hospital, "We were so crowded and flow was so backed up that our emergency department was on ambulance diversion for 60 hours a month. On average, that means for 60 hours a month, our emergency department was closed to the sickest patients in our community. One month we hit over 200 hours of diversions. We couldn't keep nurses. It was such a hard place to work that nurses would quit. We relied on temporary nurses, on agencies for placing nurses or traveler nurses to fill in the gaps in staffing." He described how the president of the hospital recognized how bad things were, and she put together a committee for flow, and he was lucky enough to be put on that committee and described the incredible transformation that occurred. Within one year, they had basically eliminated ambulance diversion. They went from 60 hours a month of ambulance diversion to 45 minutes a month. (01:23:19): They improved the length of stay for all admitted patients shortening the time that patients spent in the emergency department. They virtually eliminated patients who left the department without being seen because the waits were too long. And we did this in a time when we had record volumes, record ambulance traffic, and record admissions. He said, "We took better care of patients. It was safer, and it felt so much easier to take care of patients. It was such an amazing turnaround. In fact, that we were able to stop hiring temporary nurses. We were able to fill our staff completely with dedicated emergency nurses who were qualified to work there. In fact, our department became the number one place for emergency nurses to want to work in the Portland-Vancouver area." He said, "I had never been a part of anything that amazing before. And I haven't been since we made patient care better for tens of thousands of patients and we made life better for hundreds of healthcare workers in our hospital." (01:24:09): He described how he used the theory constraints after reading the book, The Goal and the problems and solutions will sound so familiar to anyone in the DevOps community, everything from prioritization to silo culture, to the role of leaders. And I'll put a link to the talk in the show notes. Lastly, number four, I had mentioned in the very beginning of the interview about the state of DevOps research showing up in the S1 filing for GitLab. So the S1 paperwork is what you file to go public with the security exchange commission. In order to go public, you must file an S1 before the security can be listed form S1 requires companies to provide information on the planned use of capital proceeds, detail on the current business model and competition, and provide a brief prospectus on the plan security itself. I will put a link in the show notes to a tweet by Kelly Shortridge, where she was breaking down the GitLab S1 that was posted on September 17th, 2021. And in there are the door metrics. Our four key metrics that the industry is widely adopted, and that Gitlab supports two over the four metrics, deployment frequency, and lead time for changes and provides temporal dashboards to identify the behavior of metrics over time. That was pretty wild. And it's something that I certainly never would've predicted when we were working on the state of DevOps research. Okay. Back to the interview. Is there anything else that you want to share? Dr. Nicole Forsgren (01:25:35): Just that I'm so excited. I was able to join second edition. It's an incredible book. I was so excited to see all the updates that we got to add and the case studies are incredible. And we even have a couple of extra in the online materials. I know there's one that we got added from the Navy. Listen, there was too much good stuff to add. We had too much. I would say that's probably my second favorite because listen, the Navy is doing things that other people are not doing. So we've got some really, really exciting things. Gene Kim (01:26:10): Congratulations on getting that case study by the way. Dr. Nicole Forsgren (01:26:14): Thank you. It was... [crosstalk 01:26:15] I had too many really fun, really exciting conversations with them. They are doing some groundbreaking work. Gene Kim (01:26:21): Can you tell us about that case study? Dr. Nicole Forsgren (01:26:23): I am trying to remember which hit final because they had a few examples, but I know some of the things they're doing include, I was kind of mentioning, they've got air gap work, they've got over air updates. They are helicoptering and work. And then doing final approvals over an app with push notifications. And it's almost just this perfect example of what you can do in top secret, highly regulated, very important environments. But if you give people the space to almost think about, and it's this phrase I've been using lately, imagine a world. What if you could do something and then just give people a little bit of brain space to imagine possibilities and say, okay, but this is the tech I have. Trust me, their tech is not written about in this book. The tech is not written about a DevOps Handbook, but okay. (01:27:20): Given the tech that we have, what if we had a little bit of space to do something and they come up with the most innovative solutions and the innovative creative ways to come up with something that is absolutely within bounds. And then they present these and Bill Bonwit is director over there. And they're like, "Absolutely. Let's submit this." And goes up and then their head of all IT across the Navy was like, "Absolutely." And I think that is what really encapsulates this incredible culture. What can be possible because yes, you know what? We all have some kind of constraints, right? Everyone has some kind of constraint, no matter what. Startups are going to have less money, large corporations are going to have more money, but more regulations are more [inaudible 01:28:19] right. You're all going to have a constraint. So how do you operate within those constraints? Gene Kim (01:28:26): And I'd love the constraints on a U.S. Navy ship. You are often disconnected. You are physically remote. [crosstalk 01:28:37]. Dr. Nicole Forsgren (01:28:37): You're probably dealing with 30 year old hardware on an operating system that doesn't exist anywhere else, sometimes. Gene Kim (01:28:44): Oh, Flipping awesome. Nicole, thank you so much. I will make sure that we get you on The Idealcast to do a single topic interview of things that we care so much about, but I am so delighted that we are able to get you today. So thank you. Thank you, Nicole. Dr. Nicole Forsgren (01:28:58): Thank you so much, Gene. Gene Kim (01:29:06): And that is our show. Thank you for listening. For updates on new episodes and the line up for this year's season, please go to itrevolution.com and sign up for our newsletter. We have super big plans for the remainder of the year and I'm so excited to bring them to you. The Idealcast is produced by IT Revolution where our goal is to help technology leaders succeed and their organization's win. Through books, events, podcasts, and research.