Watch and read this 90 minute Ask Me Anything with Gene Kim, author of The Unicorn Project, from January 15, 2020. Recorded for the IT Revolution Book Club.
1.Taylan Ayken & Andy Tinkham (jump to question in video) — Why Dr. Eric referring to people as “sensei”? Why does Erik forgot names? What’s the reasoning behind that choice?
2. Keith Klundt (jump to question in video) — Perhaps an observation, but I’ll try to turn it into a question as well: much of what I read in both TPP and TUP, as well as DORA/IT Revolution publications does not explicitly address team structure, at least not in the explicit way that Project to Product, Team Topologies, etc., does. In my experience, that allows many engineering leaders who express love for DevOps but still adhere to matrix org structures believe that org structure is not a contributor to team/org success. I had lunch with a VP today who said as much, i.e., a matrix org in which dev managers allocate team members to different projects is as effective as a dedicated, autonomous product team org. I would love to see concepts from, e.g., Project to Product and Team Topologies be made more explicit in DORA/IT Revolution publications. Does @Gene Kim have an opinion on this?
3. Taylan Ayken (jump to question in video) — One game I was playing while reading The Phoenix Project was associating the characters with coworkers. Writers usually base characters in their book on real people they’ve interacted with. I’m wondering who was the real Brent or Maxine or Dr. Eric?
4. Mike Snyder (jump to question in video) — I’m wondering if anyone has done a similar evolution of the DataHub transforms and Narwhal. Would you be willing to help me get the basics of the architectural approach you applied? I have a client that could use some help understanding some different basic approaches to modernizing how their data is integrated and made available for the business to use in new ways without just subscribing to a massive ETL and consolidation effort.
5. Per Krarup (jump to question in video) — We (the rebels) were inspired by The Phoenix Project to do continual deployments on physical products, inside our full scale manufacturing plant. Do you know of anyone who is doing, have tried or considered something similar? It would be nice to trade experience. — Our context: The company has a billion dollar revenue, and everyone else are doing changes, through Stage Gate projects on an annual release cycle, with a minimum of six month deployment lead time. The production start date is locked two or three years in advance, prototypes have been replaced with massive Design Review Boards. Imagine the carnage at every production start
6. Rainer Hansen (jump to question in video) — Hi @Gene Kim, I’m wondering about the ways an architect can increase flow. May you give some examples? What is the most important skill of an architect to increase flow?
7. Roman Pickl (jump to question in video) — What are your thoughts on developer productivity teams?
8. Steve Elgan (jump to question in video) — Gene, what do you suppose the origin of the ancient, powerful order (aka the bureaucracy, aka silos) is given that it exists in so many organizations?
9. Jerreck (jump to question in video) — Part 2 kicks off with the announcement of the one month feature freeze for Parts Unlimited which we saw in TPP was a huge success. I’ve read about the successful feature freezes with LinkedIn’s Inversion and Microsoft’s freeze for SQL Server security updates in the early 00s, but I was wondering if yal knew about some unsuccessful feature freezes and what might have characterized those vs the successful ones?
10. Denver Martin (jump to question in video) — What are some ways you would recommend to convert Executive Management to the rebellion? I think it is easier to find the engineers and developers and not hard to convince them to work towards the greater good. I think it is harder to find the Executives that could help, what are some ways you think we can approach those as we start seeing positive movement of the needle?
11. Proctor (jump to question in video) — Having looked across the industry are there any signs/signals that represent a likely show-stopper for transformations? What are the lead dominos that have to fall, before one can pull off the mindset change. Accelerate and The State of DevOps Reports give a number of items involved, but are there things that are the linchpins to the transformation being successful or unsuccessful?
12. Jess M. (jump to question in video) — Do you have any suggestions on ways to quantify value of having a feature freeze, or after a feature freeze quantifying the value of what was accomplished? How about any suggestions on typical/general items to prioritize first for a feature freeze that would make the biggest difference?
13. Adam Hawkins (jump to question in video) — Follow up on my previous question regarding distinguished engineer. I think asking what about org size and distinguished engineers was the wrong framing. Allow me to retry: what are the inflection points in an organization where considering a distinguished engineer role makes sense?
14. Phillippe Guenet (jump to question in video) — To what measure would you agree that Organisation and Architecture are a mirror of each other. This makes it difficult that if you want to address either you have to address both. Tech Debt needs to be addressed with the Operational and Organisational debts that go with it. — Same question for accelerating DevOps with rearchitecting in Monolithic environments. Getting the code of 200 devs to merge and build without issues is difficult. Breaking down into smaller services / smaller teams is a way of accelerating the devops practices in my view (and a path to Microservices too). Is there a max team size to Service that you would recommend?
15. Scott Styles (jump to question in video) — What do you think the geographic distribution of devops maturity is in the US? I’m located in the Midwest, but find most of the top performing organizations I hear about are on the coasts. Any tips on finding local elite or high performing organizations to learn from?
16. Scott Styles (jump to question in video) — As a highly introverted person, I find remote-first as an IT strategy extremely appealing. How does it change the Devops formula? Any suggestions on successfully meshing the two?
17. Scott Styles (jump to question in video) — Any advice on making a low performing devops organization appealing to individual contributors with experience from high performing organizations? Closing the gap from behind, when the leaders are also more skilled at learning, is extremely difficult.
18. David P Moore (Jump to question in video) — Can Gene speak more about the principle of Locality? That’s a new term to me and I’m wondering where it came from and where I can learn more about it in terms of software. Finished the book BTW and really enjoyed it.
19. Jess M (jump to question in video) — Can you help suggest ways to stop rewarding heroics? In the book we see people working overtime and performing heroics to fix problems, get releases in, etc. It starts to become a pattern and what gets recognition. Any thing you’ve seen that could work to reward no issues, non event releases?
20. Pete Franklin (jump to question in video) — @Gene Kim Maxine was awarded the “Lifetime Achievement Award to Maxine Chambers for Abolishing TEP-LARB,” and I’ve seen enough dysfunctional Technical Governance structures to cheer her along with everyone else. But the LARB was set up with honourable objectives, so if not through an ARB, how are those objectives best achieved?
21. Fernando Bitti (jump to question in video) — Hey Gene! As you are an advocate for functional programming and it is clear how it is better in most scenarios, my question is: are there any exceptions? Meaning, have you seen situations where you’d recommend solving with imperative programming instead, because it would either be faster or take less memory than with functional programming, and if so, which one?
22. Proctor (jump to question in video) — Where does one find an Erik? Wondering how does one get progress at the grassroots level when one doesn’t have the air cover/backing at the executive level, like Maxine and Bill had with Erik covering them with Steve and Dick.
Thanks for coming out to the second book club AMA. We are going to make this really short and just get started with it. We’ve got about 24 questions already. We think that we should have enough time to get through all of those. Gene has got a stopwatch and feel free to post any additional ones in the Q and A or in the AMA channel, and then I will keep adding those to the queue. Let us get started.
Yeah, and to be clear, you have 26 questions and so we do have room for four more. All right. By the way, just to reflect on last time, I was dazzled by all the questions and the breast of ground that we covered. Thank you for joining again this week. All right, question time. Starting the stopwatch. Oops, hang on. Let me arrange my windows so that I can see both. All right. Okay.
Question number one, Taylan Ayken & Andy Tinkham, “Why does Dr. Eric refer to people as sensei? Why does Eric forget names? What is the reasoning behind that choice?”
That’s interesting. Okay. First off, like some people asked, “Why did Eric suddenly start calling people sensei?” Would you believe that Eric actually did that in The Unicorn Project, but it was edited out during developmental editing phase and I didn’t know that. When I discovered that Eric stopped calling people sensei in The Phoenix Project was actually when I picked up the hard cover book. I was horrified. Anyway, we never did go change it back in just because it was too late. It was important to me that Eric do call people sensei in The Unicorn Project, and these are people I hold in very high regard. Just like when people have a PhD in general, I like to call them by Doctor or whatever was Dr. Nicole Forsgren, Dr. Topo Pal, Dr. Mik Kersten.
Just because I know how much work they put in to get their PhD is something I admire and respect. In a similar way, the people that Eric holds in high esteem they get elevated, venerated to the sensei thing. It’s just an indication that they’re held at a very high level, like Eric’s honorary PhD. Why does Eric forget names? In The Phoenix Project, Eric forgets anyone’s name, who doesn’t have the epiphany, aha moment hasn’t become enlightened. I think in particular, like John, after he has a transformative moment when he comes back having lost 20 pounds and shaved his head and he’s wearing his snazzy vest. Eric always forgot his name, but after that moment then John is one of the few people that Eric calls by their correct names.
I think in the manuscript, there’s a very specific point where for each person, whether their names are remembered. Anyway, that’s the answer. Okay. Two minutes, 30 seconds. Great question. By the way, just a little reflection as the author, I’m always surprised by how quickly people pick up on those things and how people deconstruct The Phoenix Project and I’ve always… Maybe when you asked that question, it was surprising to me that no one had ever written about that before. All right.
Question number two, Keith Klundt, “Perhaps an observation, but I’ll try to turn it into a question as well: much of what I read in both The Phoenix Project and The Unicorn Project, as well as DORA/IT Revolution publications does not explicitly address team structure, at least not in the explicit way that Project to Product, Team Topologies, etc., does. In my experience, that allows many engineering leaders who express love for DevOps but still adhere to matrix org structures believe that org structure is not a contributor to team/org success. I had lunch with a VP today who said as much, i.e., a matrix org in which Devmanagers allocate team members to different projects is as effective as a dedicated, autonomous product team org. I would love to see concepts from, e.g., Project to Product and Team Topologies be made more explicit in DORA/IT Revolution publications. Does Gene Kim have an opinion on this?”
Yeah, for sure actually. Project to Product and Team Typologies are IT Rev publications, but I guess… Does that… Interesting. Yeah, the Project to Product model never really made it into DORA and Accelerate. That’s actually surprising to me. I have this recollection that they were, but maybe that sounds like a good… Let me go study Accelerate and if we never had a question on that, I would actually be a little bit surprised. But if that’s the case then that is definitely a great candidate for a 2020 DORA.
Alex, if you could add that to our list of actions to take and I’ll make sure to pass it on to Nicole and team. Great. One minute, 50 seconds.
Question number three, Taylan again. “One game I was playing while reading The Phoenix Project was associating the characters with coworkers. Writers usually base characters in their book on real people they’ve interacted with. I’m wondering who was the real Brent or Maxine or Dr. Eric?”
Interesting. Then Alex, you might want to mute, I can hear your keyboard. Yes. I would say almost all the characters in The Phoenix Project were a synthesis of people that I’ve run into in my journey. There was specifically a specific artifact document I kept for almost 15 years.
Let’s start off on a Palm Pilot III, moved to my Treo, moved to my Blackberry, moved to my iPhone. It was called a quote file. Whenever I heard something really interesting, funny, asinine or unbelievable I would put in the quote file and then in the development of The Phoenix Project, I would put it into the… Really, the characters were a vehicle to deliver the lines that were in the quote file. Bill was definitely a synthesis of characters that I admired in people leading these high performing technology organizations. Certainly Wes and Patty were, Wes was the embodiment of all the people who would say no to things and would express things, express why they wouldn’t think we should do something.
Patty was very much the idealist and was definitely about process, whether it was idle processes or Kanban queuing processes, and Eric was very much the model up for Jonah in the goal. I think there was a question that we’ll tackle later on about mentors. Eric is not a mentor, Eric is a… Because ideally you wouldn’t want a mentor like Eric, you want people who are actually willing to go more out of their way to help you and nothing to do on scavenger hunts, although that was helpful to Bill. Brent. Brent is actually one of the few people who was actually based on a very specific person and we didn’t rename them. Brent was actually a colleague of Kevin Behr, one of my coauthors of The Phoenix Project.
Yeah. He was definitely the person who no outage could be fixed without Brent. No major piece of work should be done without Brent. His name is Brent Schultz. He worked for the CSA, Symantec. He’s often doing great things. I actually ran into him five years ago in person and I was giving a talk about The Phoenix Project in Seattle. Brent’s doing great. Maxine, Maxine is very much informed by my love of development. Actually I’m surprised I haven’t shared this. For those of you who haven’t heard, and you’ve probably seen this in the talks I gave, for 25 years I’ve self-identified as an Ops person despite getting my graduate degree in Compiler Design and High Speed Networking.
The reason why I love Ops so much is that it was my observation that Ops was where the saves are made. Ops saves the customers from terrible developers who blew up everything in production. Ops actually was the ones who secure things because despite ineffective security people and then learning Clojure, this functional programming language that runs on the JVM is what really reintroduced the joy of coding to me and so much of Maxine was informed by just the sensibilities that I learned through that experience. So much of Maxine’s beliefs were very much modeled after people like Mike Nygard who wrote the Release It! book, Cornelia Davis who was at Pivotal, Rich Hickey, the creator of the functional of Clojure, the functional programming language.
Maxine is really meant to model their experiences, sensibilities, beliefs and philosophies. Yeah. Then the joy that she gets is very much modeled after my own experience, really rediscovering the joy of coding and building things. Quick question, four minutes, 40 seconds.
Question number four, Mike Snyder, “I’m wondering if anyone has done a similar evolution of the DataHub transforms and Narwhal. Would you be willing to help me get the basics of the architectural approach you applied? I have a client that could use some help understanding some different basic approaches to modernizing how their data is integrated and made available for the business to use in new ways without just subscribing to a massive ETL and consolidation effort.”
Yeah, that one’s easy. That one is very much based on the work that Heather Mickman did when she was the senior director of development at Target. This is what she talked about. Heather Mickman and Ross Clanton spoke at DevOps Enterprise for four years, 14 15, 16 three years in a row. This is based on Heather Mickman’s, what they call, the API Enablement Project. The specific problem that they set out to solve was that every time a new Dev team wanted access to data that they didn’t have. Systems record, pricing, promotion, inventory, store information, they would have to wait six to nine months for the integration teams to create the integrations, everything was point to point.
They created this next generation system of records. That’s not quite true. They made this architectural change so that it was a proxy for all the systems record, where they put everything there and they did all the normalization, they economicalized everything so that business units could just use at on demand versioned APIs out of the box. That was running on Cassandra and yeah. I would recommend you watch their videos that’s cited in the DevOps Handbook and specifically I think it was the 2015 presentation. Basically, Alex, why don’t you send out the link for all the Heather Mickman presentations from Target.
She actually presented last year in Vegas and she’s doing the same thing at Optum, the world’s largest healthcare company, which is freaking amazing and is basically I think an order of magnitude or two orders of magnitude larger in scope. By the way, Heather Mickman actually by the time she left Target I think she had 170 engineers working for her. 53 business initiatives were credited and enabled through that effort and by now in 2020, there’s hundreds of initiatives that were enabled by that. Ship to store, a bunch of in store applications for Target employees. Actually I can’t remember all of them off the top of my head, but basically, what major business initiative could you…
All of them needed it and that was the best way to get that information. Alex will post those videos. Incidentally, Alex, one more action item, there’s a person who just tweeted at me last week, I need to get to, he’s actually presenting on something similar at I think the Philadelphia DevOps meetup. I actually want to go watch that presentation because apparently that’s exactly their area of expertise. One more thing. Alex, I’ll send you the link of the presentations of Ross Mason. He was a co-founder of MuleSoft. They’re there in the API business and his presentations are amazing. It’s about really how do you create the API ecosystem and essentially do exactly what’s described in The Unicorn Project and create discoverability so that people can just serendipitously discover what data is within the organization. Great stuff.
Anyway, I would also recommend you see the Optum presentation as well. Great stuff that’s probably definitely worth a blog post and keep me posted on that journey. In fact, if you want to present at DevOps Enterprise on that, let me know or just submit to the CFP. All right, next question. Wait, did I answer all of that? Cool.
Question number five, Per Krarup, “We (the rebels) were inspired by The Phoenix Project to do continual deployments on physical products, inside our full scale manufacturing plant. Do you know of anyone who is doing, have tried or considered something similar? It would be nice to trade experience. — Our context: The company has a billion dollar revenue, and everyone else are doing changes, through Stage Gate projects on an annual release cycle, with a minimum of six month deployment lead time. The production start date is locked two or three years in advance, prototypes have been replaced with massive Design Review Boards. Imagine the change at every production start”
Yeah, I think, Scott Prugh who I mentioned in the last Ask Me Anything is doing something similar because CSG is the nation’s largest bill printing company and so it would surprise me if… It was in the scope responsibilities, are those first of all for the plant control systems?
Alex, let me queue that up for next session and let me think about that a little bit more. That one deserves a better response and it’s just elusive to me right now. I’ll get back to you on that Per and let me set your expectations, I’ll give you some concrete company names, names of people and ideally the presentations. To be continued.
Question number six, Rainer, “Hi. I’m wondering about the ways an architect can increase flow. Can you give me some examples? What is the most important skill of an architect to increase flow?”
That’s interesting. Last time, I’m pretty sure I had referred people to Dr. Mik Kersten’s presentation, the last plenary presentation that he gave, which I think was in London or was it in Vegas?
Actually, no, it was in Vegas 2020 but it wasn’t plenary. If you listen to the last 10 minutes of it, he specifically talks, he gives like three examples of that, of flow problems and how they fixed it. Let me see if I can recall help with them. One example that he gave was he and they were specifically looking for wait states in terms of why aren’t developers more productive? Typically, because they’re waiting on things. In one particular case they had a product and these are all internal case studies, which was just even better because he was presenting all the information on it.
One was a for a specific product where they had a month integration testing cycles and so they spent I think a quarter basically re-platforming that so that they could actually test within minutes or hours as opposed to weeks or months. That dramatically improved flow and they went from the most unhappy team in the company to the most happy team. There was another one that was about… They had switched from Java to Scala and the goal was actually make it easier for developers to be productive and I think over a couple of intervals they were actually really worried that maybe it wasn’t working, that maybe… The quote he used was “Maybe our developers aren’t smart enough to use Scala.”
They were about ready to yank that initiative back and the team leads like, “No, no, we got this, we just need another interval or two.” Then they actually did see the productivity gains that they were expecting. That was awesome. In this case it was the language, I think it was a porch, but they were doing new initiatives in a new language to help them get better at it. A third one was… I think there might have been one where there was certain things that were cross cutting concerns like the canonicalization of worker item IDs and it meant that every team was taking on the load of having to, whether it was serialize or economicalize or whatever.
I don’t remember the exact details, but the architectural changes then moved to a central function so that because it was a cross cutting concern, one team could focus on it and then everyone else would benefit from that work. Yeah, those are very specific examples. I would call the obidos refactoring at Amazon, a famous one where they moved everything to firm API boundaries. What are the skills of great architects? I think it’s the sensibilities of how to decomplex things in a way to use the Rich Hickey language. It’s the things that are intertwined, how to separate them so that A doesn’t know about B and vice versa because only then can things be decoupled and that seems to be one thing that architects are really good at.
I think another thing is creating these platforms that enable developers to be productive. In other words, platforms or anything that has a crosscutting concern, how do you remove that responsibility from the teams and move that into a specialized area so only one team has to worry about it. I think those are really all about taking a system’s view. These kinds of sensibilities of how to separate things. Then I think another thing is maybe just knowing what’s possible, because I think these people are very good at studying what other people have done, understand what the patterns are and then seeing how those could be applicable to the problems that you face.
That’s maybe not the best answer, but I think the best person the game is definitely Dr. Mik Kersten. I would just recommend any presentation that he gives. Okay. Four minutes, 40 seconds. Maybe a little bit off pace, Alex, but I’m not worried yet. The problem is these are fun questions.
Question number seven, Roman Pickle, “What are your thoughts on developed productivity teams?”
Yeah. Back to the previous questions. Aha moments was this inversion of values and I love this model where there’s these three categories of work. You have the applications, and I think I talked about this last time, the features. Anyone can budget for those because everyone sees it, they see the app, they see the feature.
In general, in most organizations they put their best developers there, the next tier down are the systems of record, back-end systems. The people who aren’t good enough to work on features, they get put to work there. Then the last tier is Dev productivity, which is where you put the summer interns. They’re the people who build and maintain the Jenkins Pipelines and so forth. What I’m just dazzled by is in the tech giants, Facebook, Amazon, Netflix, Google, it’s the exact opposite. They put their best people on Dev productivity. Then it’s the common platforms and then if you’re not good enough for those two, you work in feature teams.
The best marker of this is that if you look at where all the PhDs and static code analysis, programming languages, they are in that top tier. One of the very few places where you go work is at Google and Microsoft, Facebook, Amazon, Kent Beck was in that category who founded Extreme Programming, helped create Agile and so forth. That’s the kind of work that he was doing at Facebook. I think those are the teams that create and enable every other developer to become more productive. I think they just have like the architecture; they have the sensibilities of what is needed to make development productive. In fact, I think one of my favorite quotes in the DevOps Handbook was from… Was it from Kent Beck?
Facebook famously deployed every day at 02:00 PM. Whatever was in master would go through some sort of test cycle and they would do a push at 2:00 PM and they did at 02:00 PM Pacific time because everyone was in the office. So if something went terribly wrong, there was still time to redo things and everyone is still there before they left at five. Then as they created the London office, it was actually very inconvenient for them because they would have to wait a day to push their code. So they ended up doing two production pushes per day, and I think that was very much informed by Chuck Rossi. I think he was director of operation at Facebook and Kent Beck even did an article about how they needed to actually do two production pushes.
I don’t even know how many production pushes they do per day now. Anyway, it’s not about just platforms, it’s about these systemic issues of how does one to do a deployment or how is work done? It goes beyond just the deployment tools but about just how the entire organization works. Hopefully that’s helpful.
Question number eight, Steve, “Gene, what do you suppose the origin of the ancient powerful order to bureaucracy silos is given that exists in so many organizations?”
Great question. I was just talking about this with Dr. Steven Spear, a mentor of mine yesterday, and one of my favorite books that I know I’ve blogged about is The Other Side of Innovation by Dr. Govindarajan and Dr. Trimble.
The two professors at Dartmouth University in the business school. They ask something similar in a slightly different frame, which is they asked, “Why do innovation efforts fail in most organizations?” Their answer was that is because of these very powerful bureaucracies. The reason they gave is that when an organization become very successful, in fact these organiza… What creates greatness in their mind? Greatness is multimillion dollars of revenue that are sustained over decades, and they say that it’s because it create a performance engine. The performance engine is what sustains greatness over time.
That means in product development, in supply chain, in sales and marketing, this is when you create processes and bureaucracies, because they’re very resilient. You can take out half the bureaucrats and the processes still survive, which is the definition of resilience. They say the reason why innovation efforts fail is because everything they’re trying to do is actually against everything that the innovation engine is trying to do. The performance engine will impede it. They basically create the notion of a dedicated team. They’re separated from the performance engine, held to a different set of rules and standards and regulations and so forth. It’s led by someone who has respect from the performance engine.
Who’s probably has done lots of time in the performance engine and that person, everyone knows has the best long term interests of the organization at heart. This book studies where innovation effort succeeded. It was the BMW electric car project, the first trail running shoe at Timberline, the first digital operation at Wall Street Journal. It was the first small tractor line at John Deere. I think they just did a great job in really explaining why performance engines exists, why they’re organized that way, what are the characteristics of them and I think the same thing that’s happening in technology. Maybe not with the great outcomes, but I think it certainly does explain why things are organized in the way they do.
Maybe I’ll just want to mention one more thing. I just came from Courtney Kissler, she’s on the program committee for DevOps Enterprise. She’s did time at Starbucks, at Nordstrom, at Nike and it’s actually, I learned from Dr. Steven Spear and his MIT workshop is just two ways to organize. You can optimize for cost or you can optimize for speed. Optimizing for cost is where you end up with functional silos. That’s also where you optimize their skill. When you have very deep expertise areas, which is why they do that in aircraft carriers.
You want all the aviation people to be in one place, all the ordinance people to be in another place, all the cooks and meal prep people in another place because there’s not a lot of value in cross functional teams there because you really need expertise to be honed. Then you have optimizing for speed, which is where you have these cross functional teams. I think that is actually pretty well written up in the DevOps Handbook and I think I might have some citations there. Hopefully that gives you some pointers on where engine powerful orders come from. The rebellion is definitely choosing to optimize for speed.
I love that quote and I actually find the prognosis of the quote already in the resource guide for The Unicorn Project. “It’s not big beats small, it’s fast beats slow.” That came from Rupert Murdoch and I think I heard it first through Chris O’Malley, the CEO of Compuware. Anyway, I can’t wait to read because I think that’s definitely the frame for the age of software, it is all about speed.
Question number nine, Jerreck, “Part 2 kicks off with the announcement of the one month feature freeze for Parts Unlimited which we saw in The Phoenix Project was a huge success. I’ve read about the successful feature freezes with LinkedIn’s Inversion and Microsoft’s freeze for SQL Server security updates in the early 00s, but I was wondering if you all knew about some unsuccessful feature freezes and what might have characterized those vs the successful ones?”
Boy, that is a great question. You’re pointing out some of the survivor bias. It’s like we only hear about the survivors and not the one that didn’t survive. Yes, there is one. In fact, this is so good. I’m going to look for it right now. There was actually one that was amazing and I love this more, even more because it’s about something that I’ve written about before, which is Etsy Sprouter, which is one of my favorite case studies about the birth and death of Sprouter at Etsy, which is this middleware they put in Devs on the front end inside of PHP and the DBAs on the back-end who are working inside of Postgres stored procedures.
That was this incredible problem because it just meant that you had to have three teams to communicate, coordinate Marshall sequence, de-conflict and so forth for anything to get done and we learnt horrendous outcomes where basically every deployment became a mini outage and when you’re doing 30 deployments a day, that’s a real problem. That presentation about killing Sprouter by, I think it was Ross Snyder was I think one of the best presentations I’ve seen. It turns out that there’s someone who actually wrote extensively about their attempt to kill Sprouter that came before it. Let me just switch Slack channels. I’m going to find this, because it’s just that good. Hang on. I store all my tweets and stars inside of…
Because searching Twitter is so hard, I store everyone in Slack because Slack is so good at searching. Man, yeah. Wow. I’m pulling it up right now. This is amazing. Here is a tweet from Dan McKinley who actually is a Clojure fan and he was part of the first attempt to replace Sprouter, which is just incredible. Twitter can be so amazing because you get to serendipitously discover gems like this. Alex, I’m just going to DM this to you, if you could just send that out. This thread talks about this ambitious effort to play Sprouter that just didn’t work and he goes into the horrible, horrible things that went wrong. That happened.
There was definitely a will to solve the problem, but maybe it wasn’t skill or something that they did wrong. I think this is awesome. In fact, Alex, maybe cue this up as an action. I’d love to actually write about this in a blog article and to the person who asked this, if you could read this and give your impressions and post it somewhere in a Slack message or in a tweet with your synthesis of it, I think that would be amazing and just mention Alex and me and I’d love to actually see what you write up. Great question. Fun. By the way, Dan McKinley is brilliant, just a top notch thinker. That was three minutes, 30 seconds.
Question number ten, Denver, “What are some ways you would recommend to convert Executive Management to the rebellion? I think it is easier to find the engineers and developers and not hard to convince them to work towards the greater good. I think it is harder to find the Executives that could help, what are some ways you think we can approach those as we start seeing positive movement of the needle?”
This has been one of my top aspirations for the DevOps Enterprise Summit is in fact we have a whole track called Spanning the Business and Technology divide where we are looking for business leaders to go on stage with their technology leader counterpart and talk about how all their dreams and aspirations and goals could not have been achieved if it weren’t for these incredible rebel leaders who helped them achieve their goals.
We’re not looking for people to just acknowledge knowledgeably we’re looking for people to gush and endlessly thank those people who have a sense of gratitude and appreciation, who tirelessly helped that technology leader remove obstacles and find more money and do all the things that need to be done. I would say anytime you see more than one person present on the plenary stage at DevOps Enterprise in general, those tend to be business leader, technology leader presentations. In fact, Alex, maybe a question to you is, do we tag the tracks of the plenary presentations? If we do, then maybe that’s an easy way to pull them out. Anyway. My advice is look at any of those presentations.
If they’re on the plenary stage and they are a business leader and a technology leader… I’ll give you one. One of them was Scott Prugh presentation in CSG that was given with his product counterpart. Which one was that? That was such a great presentation. I think that was in 2018. Another… Alex, I didn’t come up with a list. Let’s work on a list together of those presentations because those are the most exciting presentations for me. Another one is Werner Loots from U.S. Bank that he presented with an engineering leader and his boss. He came from a strategy background. I think he came from McKinsey. He’s an executive vice president in charge of all of consumer transformation.
Another great one was Comcast last year in Vegas. The business leader wasn’t there, but they actually talked about how that team was acknowledged by the CEO, I think John Roberts. I think these are great presentations that show how teams earn the respect of their business counterpart. Alex, why don’t you and I take the action item to create a curated list of just the spanning the business technology divide presentations and then to concretely. One, I think you should watch that. Then two is I would send that video out to prospective business rebellion sympathizers and as I say, hey here’s a way that… Another one was Capital One.
John Schmidt was the director of product strategy at Capital One, co-presenting with Aimee Bechtle who was part of the DevOps Dojo. Basically he’s talking about how all their machine learning initiatives, we’re in trouble and it was actually through the Dojo’s help that they were able to achieve their goals. It is a phenomenal presentation and they’re giving from the business perspective. Whenever you see those present… I would send a couple of those links out to them and just say, “Hey, does this resonate with you? I would love to brainstorm about ways for us to do something similar at our organization.” And I love it because what it should do is say, “Hey, the person on stage is like me.”
And you are the person like the technology leader, right? I think it just gives a very concrete way of saying, “When would you like to do this together to co-create something?” I would also put in there any panel that I’ve done with Chris O’Malley, the CEO Compuware. I love the one that he did with the CFO, Joe Fe. These are all intended and designed to speak to business leadership, to help sponsor better ways of working. Maybe the Jon Smart one also goes there, with what he did with the director of project management. Great. Great question, and please keep me posted. We have some action items. Alex and I will create a list of videos for you and I would love a report out on how it went and keep me posted. I’d love to help any way I can.
I see a chat. Brian Clark, thank you. Brian Clark was from CSG that co-presented with Scott Prugh. Thank you, Mark Fuller. I thought that was Alex. Awesome. Yes. He was the VP of product. The profit and loss owner for an $800 million line of business for a publicly traded company.
Question number eleven, Proctor, good hearing from you again. “Having looked across the industry are there any signs/signals that represent a likely show-stopper for transformations? What are the lead dominos that have to fall, before one can pull off the mindset change. Accelerate and The State of DevOps Reports give a number of items involved, but are there things that are the linchpins to the transformation being successful or unsuccessful?”
Great question. I think it’s very much similar to what the previous question was, I think. The reason why the spanning that business technology divide is such a focus area for us in the DevOps Enterprise Program Committee is because that is the obstacle now. In order for these transformation to really succeed, digital disruption, whatever you call it, those are not technology transformation issues. It’s a business transformation issue, and for that we need business leadership fully on board. I think one of the best statistics on this came from a McKinsey study that came out last year and this was cited in Mik Kersten’s presentation that over a trillion dollars would have been spent in 2019 to do digital disruption, and 90% of them have no monetary return whatsoever.
I think the number is 1.2 trillion, whatever it is. The number comes up to $850 billion of wasted effort. It says there’s demand for this, whatever is spawning that, there’s an appetite to do these changes. I think one of the necessary conditions is for someone at the top to be thinking about this and as Chris O’Malley from Compuware said, “There’s no board on the planet or a fortune 500 company that isn’t thinking about it.” That’s the good news, but now we’ve got to find someone like Werner Loots the EVP at U.S Bank, to fully sponsor it. Just to hear him talk about the business imperative, that started off as a focused effort when Minneapolis hosted the Super Bowl.
They had this crash program around how do we make ourselves visible in the Super Bowl and how do we build things quickly? That really sustained itself even after that Super Bowl. It was like, “All right, this is going to change how we work. We want to reimagine ways to deliver value to our customers.” They created a dedicated team, totally different fore space. Very much alluded to in the other side of innovation. Then I think that really sets the stage where you have this incredible initiative that there’s no doubt in Werner’s mind when you hear him talk that his success is totally dependent on… I can see his face. What’s his name? And just to see how they talk and interact with each other is just awesome.
I would say, Proctor, I would look for the most senior person on the business side who cares about these things and whether the strength of that relationship between them and their technology counterparts and yeah. I think if there is a person that that person is working with that there’s a mutually trusted relationship or even a shot at a mutually trusted relationship. I think in my mind that is the most critical precondition. Is there a digital transformation effort underway that’s being talked about? Then, who’s in charge of that on the business side and who are they working with and are they getting the help that they need to actually achieve all their goals, dreams and aspirations?
If you want to talk more about that, Proctor, you know how to reach me. Proctor is the person who’s podcast Functional Geekery. I’m just a huge fan of. It was great to be on his podcast last year. Okay.
Question number twelve, Jess, “Do you have any suggestions on ways to quantify value of having a feature freeze, or after a feature freeze quantifying the value of what was accomplished? How about any suggestions on typical/general items to prioritize first for a feature freeze that would make the biggest difference?”
I can give you one that is… That’s a great question. Let me take a drink of water and think about that for a moment. I’ll tell you what it can’t be, it can’t be features because specifically we’re talking about like, we’re not working on features, but we’re going to deliver something else. The flow framework by Mik Kersten gives us the frame of what the answers have to be in. It has to be, if it’s not a feature, the other three are defects, risks, and debts. By definition, we have to be able to articulate the value in our ability to reduce the number of defects, to increase our ability to deliver services reliably, securely, etc. The risks we need to mitigate and then reduce our technical debt. Then we measure technical debt by the opposite of technical debt is architecture. Our ability to increase flow. Right? To allude to the previous answers.
Everything really should be, at least in this frame, is how much are we able to increase developer productivity? Meaning, can they do their work without toil to reduce the wait states by reducing the dependencies with other teams, whether it’s other feature teams or other infrastructure teams, storage, networking, security, whatever. To what degree can we enable them to work independently so they can develop tests and deploy value to customers independently? That’s improving flow as measured by reduction of lead times, by increasing the amount of time that they’re able to work versus wait. I think in general; I think that’s the way we… I love this.
By logical rigor, those are the ways that we have to be able to talk about the value we create during a feature freeze. Let me just test that answer. In Microsoft example of the security stand down, they were able to reduce the number of systemic problems within their products across product lines and integrate security objectives into everyone’s daily work. At eBay, they measured it by their ability to actually keep their site up and also, they should be able to also measure their ability to actually have developers actually work on features again, which is actually a good thing.
LinkedIn, they were actually able to dramatically increase the rate of feature delivery and they talked about how deployments went from a three-day horrendous affair to something that just happens on demand and unleashed… In my view, two orders of magnitude of productivity as measured by features that are actually able to be appreciated by customers. Yeah. I think all of those are just some magnificent lens to measure the business value created by feature freezes. Alex, let’s turn that into a blog post. I think that’s a great one. In fact, makes me wish maybe in a second edition of Unicorn project, we can actually put that down. That sounds pretty awesome. Just a little side note.
Just like it really upset me when sensei’s were accidentally removed out of the finished project, it’s just so hard to go back in. I’m just reluctant to make big changes to anything that’s out there. There’s something about the long form writing style that makes it very difficult to change. I just feel like everything’s so tightly coupled together and you don’t really want to dramatically change the reading experience for people. Anyway, just a little observation. Actually, let me just do a little time check. Here we are at 50 minutes, Alex, is that right? Is that correct?
We’re a little over halfway.
Awesome. Okay. We’re at number 13. I think we’re on track.
Yeah, we might be a little behind on time, but we’re doing well.
Okay. Alright. No worries. Condition yellow, not condition red. By the way, could everyone just give maybe by an emoji reaction, how are we doing and let us know. Then Alex, if you can send a report out on that in the next thing. Yeah, if everyone can just answer the question, how are we doing and respond in the form of an emoji reaction, that would be great. You know what, I’m just going to pop into that channel. Sweet. Great. Thank you.
Question number thirteen, Adam, “Follow up on my previous question regarding distinguished engineer. I think asking what about org size and distinguished engineers was the wrong framing. Allow me to retry: what are the inflection points in an organization where considering a distinguished engineer role makes sense?”
That’s a great question. This is really bothering me now because I’m going to refer this present… The person who presented with Levi Geinert, who was an engineering director at Target, he was the one who is now a VP of engineering at U.S Bank who co-presented with Werner Loots at U.S. Bank. That was last year, in 2019. The year before, he presented as part of a trio with a distinguished engineer Dan Cundiff and one of the product people, Luke. It’s really interesting to hear about the creation of the DE role at Target because they talk about the session they have with, Dan called him uncle Mike, Mike McNamara the CIO of Target.
Apparently every quarter he has a multi-day offsite with all of his direct reports and the engineering directors, which I think is really interesting. Just even hearing about the format of the meeting and really talk about the weedy issues that face the organization. I think we can start to derive what are the conditions that lead to the creation of the DE role, which is that, at what point is it not enough to have just the senior managers there? It’s like if we’re talking about like how do we get away from hostile vendors and how do we create a database strategy where we’re not reliant upon vendors that we don’t want to work with or middleware companies we don’t want to work with or whatever?
At that point I think senior managers are good, but really even more valuable are domain experts, who their job is not to manage people, it’s really to have strong opinions about what technology we should and shouldn’t be using. I think what’s alluded to The Unicorn Project maybe not explicitly is yeah, as a Parts Unlimited, it was at a stage where Bill and Chris the VP of R&D and the CIO, they’re not experts in message buses or architectures or etc., and Maxine is. She’s serving a very critical role in how technology is used at Parts Unlimited, just like Dan Cundiff and the other distinguished engineers, how their job is to identify what technologies are going to be used at Parts Unlimited and it’s so important that they’re taking part in the senior leadership meetings.
That’s a great question. You know what, maybe that’s a question that Dan Cundiff should talk about at DevOps Enterprise and even if he doesn’t, that’s probably a good podcast episode or something. Alex, why don’t you take that on, let’s record that and maybe as aspiration for 2020 I’d love to get you better answers on that. It’s a great question and I think it would be valuable to the entire community to actually have a better answer for that. Alex, I think we should be having a meeting specifically about how we process all of these in the next couple of weeks. All right, great question.
Question number fourteen, Philip. Remember that great story about you using Unicorn Project declines to find fellow rebels and help grow the cause? I just love that. In fact, I was talking to Jason Cox from Disney and they’re talking about how much delight they’re getting that the rebellion meme is taking off within this community. Rebel scum. We’re joking. Okay, cool. “To what measure would you agree that organization and architecture are a mirror of each other?” Yeah, 100%. “This makes it difficult that if you want to address either you have to address both. Technical debt needs to be addressed with the operational and organizational debts that go with it.” Yes. “Same question for accelerating DevOps with rearchitecting in Monolithic environments. Getting the code of 200 devs to merge and build without issues is difficult. Breaking down into smaller services / smaller teams is a way of accelerating the devops practices in my view (and a path to Microservices too). Is there a max team size to Service that you would recommend?”
I don’t really have an opinion of that last one just because I don’t have a lot of experience, but I would look at Google and just… Let me answer it backwards. There’s a maximum size of service. In the DevOps Handbook, there’s a citation of Randy Shoup’s work. He was the engineering director at App Engine. It was a platform as a service offering like Heroku instead of Google and he actually talks about how the Google technologies are organized.
Their equivalent of S3 buckets are on top of cloud storage on top of whatever and the team sizes are actually very small given the scale of the services they operate. There’s a citation in there about how he talks about that. Then he actually has an entire talkie where he actually explains that and gives specific figures. I would refer you to that. Because I think the infrastructure that runs Google is a pretty massive thing. I would say that would probably represent a pretty accurate indicator of what the upper limit is and what team sizes are. He just makes the observation that these are services that built to one of the services so the team sizes never get too large.
To your other point is you’re pointing at the duality of organization and architecture and yeah, that is actually the big aha moment for me. That really is one of the foundations of The Unicorn Project, which is that it is not enough to move boxes around on the org chart, that the organization and the architecture have to be congruent meaning that and I think there’s almost a… It’s not an isomorphic mapping, the architecture has to support this way of working. Let me try that again. If the goal is to get teams to be able to develop, test and deploy value independently, if that’s the goal, there has to be an architecture that supports it. There’s a Conway’s law in there somewhere that…
I’ve never done this before, but I’m going to pull up a PowerPoint slide in an attempt to try to verbalize this because that question is so good. Just updated Microsoft office, but it works. Good, good, good. Alex, while I’m looking for this, can you give me a verbal readout of the emoji reactions? How are we doing?
Lots of thumbs up, unicorns, smiley faces, inquisitive face, hundreds. Looking good.
Okay, good. All right. I’m going to share my screen. I had mentioned the birth and death of Etsy Sprouter as the ultimate example of Conway’s law. They had three teams working together, which included in the terrible Sprouter thing that everyone loads and then the goal was every team should be able to work independently. One, that no team has to communicate and coordinate in order to deliver value to the feature. I wrote down this aha moment, and this was after the DevOps Handbook. I don’t think I’ve actually written this down in this way, I think this is why I’m struggling to verbalize it. The key lesson is that the organization and the architecture of our software must be congruent.
In other words, you can have a bunch of teams, but if you don’t have an architecture to support those teams to work independently, then we’re just fooling ourselves. When you have a value stream, if this is a deployment value stream, if you initiate the ticket to deploy here and you have to transit across 20 different teams to deploy, then your architecture dictates that you do all this type of work. So yes, I think I very much agree with your assertion that there must be a congruence between the desired way of working as dictated by teams, the way you organize your teams and the architecture that enables that way of working.
If you have feature teams that also work independently, but you still have to do this, then you’re actually all still coupled together. I think that is my point. If you want to discuss that further, let’s do so. In fact, because I think it’s worthy of being written down because it’s very powerful and I think the Team Topologies book even says it more verbally or maybe I think it should be in there. Just take a picture of that page and post it.
He just shared. He said, “I think they use in Team Topologies, the phrase socio-technical structures to describe the congruence of team architecture and team structure.”
Brilliant. Yeah, Philip that’s one angle and I’m sure if that were enough, you wouldn’t have asked that question. Let’s talk further. Okay. We went a little over on that one. That was six minutes, 30 seconds, but it was great.
Question number fifteen, Scott, “What do you think the geographic distribution of DevOps maturity is in the US? I’m located in the Midwest, but find most of the top performing organizations I hear about are on the coasts. Any tips on finding local elite or high performing organizations to learn from?”
That is interesting. We never did that analysis in the state of DevOps report. There was… The common wisdom, that is wrong, is that only financial services are doing this or only certain vertical, like retail are doing this. And it turns out that’s actually quite wrong. The distribution of high, medium and low performers is about the same. It has been for five years and that changed in the 2019 State of DevOps Report where retail had just operationally a high number of high performers. That’s interesting. If you’re looking for elite performers, I have two answers. I would look at the good startups in the area because really startups are, the good ones, the good startups are adopting the best known methods of deploying software like life and death depends on it, and they have an ample reason. There’s not a lot of legacy reasons to do things in an older way.
That’s not to say all startups are good. Find the ones who are doing innovative things that are growing, chances are there’s some good stuff happening there. Then also look at the DevOps Enterprise list. At the very minimum you can look at for the speakers that span all geographies, I can say it with confidence that representation of organizations is pretty even and you’ll find them all geographies. Then I wish that there were… I know we can’t share the attendee list. I know that’s not allowed, but I wonder if there’s a… If you have gone to DevOps Enterprise, pop into the DevOps Enterprise Summit Channel/Gala we have, and I would actually just ask around in there and I’ll also look for any DevOps meetups or something.
Often these are the people who are hiring. The organization are taking a very active role in community development, so forth because this is just one great way to recruit. For example, I went to just hang out with Jason Cox at Disney, they’re hosting all the Docker meetups, because they’re looking for more people with a skill in Docker. Yeah, those are some very tactical tips I would recommend. That was a little over two minutes, Alex.
Question number sixteen, Scott, “As a highly introverted person, I find remote-first as an IT strategy extremely appealing. How does it change the DevOps formula? Any suggestions on successfully meshing the two?”
A funny comment, by the way. Brad Feld recently did a post. He’s the famous investor behind Foundry Ventures. He’s one of the funds with the top returns the last 30 years. He just did a blog post, Alex, maybe you can find this. I got it through his newsletter about remote work and just how appealing it is and how much is being adopted and I think it’s just showing that remote work is more of a thing. In fact, there was an article in New York Times about this as well. GitHub, back in 2013, I think if you look at their SRE team or their infrastructure team, I think there was a point in time where no two engineers worked in the same city. It was fully distributed and they were giving these great talks on how they were adopting DevOps and amazing ways of working for fully remote teams, basically optimized for the remote experience.
Alex, why don’t you sign me up for that action, I’m to post that. In fact-
What was the name of the person one more time?
The name of the person one more time?
I think… Yeah, Kathy Kidding just posted it so we’ve got in there.
That’s awesome. This is a great channel. I love this. In fact, I should not be posting this in my DMTU, I should be posting it in the chat message so everyone else can see it. The presentation I’m thinking of is, it was at Ruby Conf in Salt Lake City in 2013 from GitHub. Who was it? He was announcing the creation of their bot, the deploy bot thing. Anyway, Alex, let’s go find that presentation and if it’s not that…. Yeah. They talk a lot about how the bot enabled people to work better remotely, take better care of each other, do deployments within the chat room. It was awesome.
The answer is remote work is definitely a thing, especially given that the search for talent, like Target talked about how basically they couldn’t find all the engineers they were looking for in the Twin cities area. Minneapolis, Saint Paul and I was actually driving a whole bunch of hiring those remote employees. It just says all the trends are working in your favor, in my opinion. Okay.
Question number 17, Scott, “Any advice on making a low performing devops organization appealing to individual contributors with experience from high performing organizations? Closing the gap from behind, when the leaders are also more skilled at learning, is extremely difficult.”
That’s an interesting question. I think what it’s making me recall is that a lot of people that I talk to, they’re always looking for challenges. In fact, the reason why people leave organizations isn’t because they hate it, it’s because they get bored. I think that happens a lot with leaders is that they get bored because they’re not challenged anymore and they recall with fondness solving tough problems. It’s actually a shame when leaders solve these great problems and then aren’t asked to solve bigger problems. Maybe that’s exagger-ness maybe it’s because the company’s not growing anymore or maybe they’re shrinking right now, that’s a whole bunch of challenges that aren’t fun anymore.
To answer your question, I think I would appeal to people in that situation where they’re looking for new challenges and I think it’s a very easy appeal to say, “Hey, do you know of anyone who’s looking for a role to help us pull off a transformation or would that include you?” The soft ask is you ask them if they have any friends or the more explicit ask is like, “Would you be interested?” I think your appeal is going to be based on not how great things are in technical practices, but what a great organization it is, the mission of the organization, how good the people are. We just need someone who’s done it before to help us get there faster. I think those make for very compelling appeals and…
Yeah. It sounds like you’re in that position. I would love to know how that goes. Let me know how it goes and whether it worked and if it didn’t work, let me know and let’s talk. Two minutes, 40 seconds.
Question number eighteen David, “Can you speak more about the principle of Locality? That’s a new term to me and I’m wondering where it came from and where I can learn more about it in terms of software. Finished the book BTW and really enjoyed it.”
Okay, great. Awesome. I would refer you to The Love Letter to Clojure thing that I wrote last year and I talk a lot about the first idea of locality and simplicity. Locality really is the ability to independently run and test your component in isolation from everything else.
You can’t have locality… Well, I suppose you can. One aspect of locality is, to what extent can you run and test things without having everything else present? Another one is just a measure of if you need to do something, can you change it in one place? One file, one module, one name space, one application, one container, one something. That’s the ideal… Not ideal is that you have to change it everywhere. You have to open up every namespace, every module, every application, make changes there. It’s scattered or better yet splattered everywhere. I think I might have talked about it last time. It was a whole thing about aspect oriented programming that Mik Kersten, he got his PhD in that. That actually showed up in Spring.
Spring incorporated, it’s a Java framework, probably one of the most used frameworks on the planet, very voraciously used aspect-oriented programming as a way… Aspect-oriented programming is you can take your cross cutting concerns and make the change in one place and then it can be used by every other component. Sidecars and Kubernetes is a way to do the same thing. Things like authentication, logging security type things, you can actually put in one container in one sidecar and then have every container benefit from that. Those are all things that encourage locality. Look at that Clojure love letter and I talk a lot about that. Gosh. That architecture question, whoever asked about architecture sensibilities.
One of the big aha moments for me learning Clojure was, and one thing that Rich Hickey talked about was the ability for cues to decouple components from each other and that was just amazing to me. I learned this, again from Scott Havens who was at Walmart, who came from Jet.com and I asked him, “How do I serve?” I take this book tracking tool that I wrote where everything was so tightly coupled together and I wanted to get away from this piece of code I wrote in Ruby to get to my SQL that was written in active record that was really hard to work with and move it to Clojure. He said, “Put it on a Pub Sub-Q.” Like a Kafka thing or whatever, “…and just post a message there and now you’ve now decoupled these two pieces from each other.”
It is amazing. Even this trivial example of this book tracking application I wrote, just putting things on cues, make things more localized so that they know nothing about each other and the only way they can talk to each other is through the Pub Sub-Q. I think people are using Kafka for things like that. I think those kinds of patterns are something that architects need to know about because these are the patterns that allow for massive decoupling of services, and that’s actually alluded to you on the very last page of The Unicorn Project before the epilogue. Anyway, I hope that helps you find more information about locality.
By the way, that Clojure love letter, I think it was 5,000 words, it was huge and I actually wrote about that one of the biggest problems I had in my vocation as a programmer is that whatever I wrote would collapse in on itself like a house of cards after a couple of years. It would just become impossible to change and writing this Love Letter to Clojure, it made me realize I was making the same mistake in graduate school. One of the projects was writing a modular three compiler that would generate assembly code for Spark. That was in graduate school and the problem is I had done something wrong in the co-generation or assembly.
I was working into stack backwards, it worked fine until we had to test it on recursive programs and then it would just blow up. Despite working, doing three all-nighters or maybe it was a week of all-nighters, I couldn’t fix it because I didn’t write them in a way where they can be composed. In other words, every successful step was buried inside of the previous step, and so you couldn’t independently run each phase without pulling all the other stuff in. I wrote that 1994 and I still remember to this day. That was a failure of composition and locality. It was very cathartic to write that and just realize that I’ve been making the same mistake for 25… Calculator, 2019 minus 1994. I’ve been making the same mistake for 25 years.
I think locality is something that one should learn early in their career. Six minutes. Holy cow. I went way over on that, but it’s just so interesting to me. I had to, sorry. All right.
Question nineteen, Jess, “Can you help suggest ways to stop rewarding heroics? In the book we see people working overtime and performing heroics to fix problems, get releases in, etc. It starts to become a pattern and what gets recognition. Any thing you’ve seen that could work to reward no issues, non event releases?”
That’s such a great question. I feel like I’ve written about this somewhere. It’s reminding me of a lot of DevOps Enterprise stories where deployments used to be horrendous and now they’re routine, and now we’re doing it multiple times a day.
It reminded me of the presentation that talks about celebrating those achievements that make it easier so people don’t have to work overtime. I think a concrete recommendation is have celebrations, to celebrate milestones in getting there. Something that used to be tedious, error prone, lengthy, that would create necessarily heroics, we celebrate their elimination of and it very much is like the hallmark of… It’s also reminded me of the birth of automated testing at Google where they did the same thing and this was a multi-year effort to basically re-integrate automated testing. It’s a way Google ships code. I think it is about…
It started with testing on the toilet, T-O-T-T, where it started off as a guerrilla movement to put flyers in the bathroom stalls just to educate people on what is automated testing, what are tips and how to do it better. That led to people pulling their 20% time whether you’re working or whatever, and people pulling that time to create guilds, I guess we would call it in today’s terminology, to make a massive improvement in the automated test infrastructure, which it was. That happened in 2005 that led to Google eventually running a 150 million automated tests per day as one of the hallmarks of their software culture.
Yeah. I think it really starts with celebrating every win and stating that eliminating heroics is one of the long-term objectives of a company because heroics in unplanned work aren’t free, they come at the expense of other planned commitments that we make. There’s a huge cost paid when we force people to do heroics. Then it sounds very much like the question around how do we measure the success and the business value of feature freezes. Because by eliminating heroics, we can actually increase feature delivery. We can decrease the number of defects, we can increase the reliable insecurity and the decrease the risks associated with running our services, and we can reduce technical debt.
Hopefully that’s helpful. Great question. Three minutes and 50 seconds, and we have 11 minutes left. I think we can do three more.
Question number twenty, Pete, “Maxine was awarded the “Lifetime Achievement Award to Abolishing TEP-LARB,” and I’ve seen enough dysfunctional Technical Governance structures to cheer her along with everyone else. But the TEP was set up with honorable objectives, so if not through an ARB, how are those objectives best achieved?”
Yes. By the way, let me show you the real TEP-LARB certificate that actually came from Heather Mickman. I’ve closed it by accident. This is the actual picture I took of Heather Mickman’s desk when I shadowed her for three days.
I think we’re seeing the wrong screen. Are you pulling a PowerPoint? Is that where you’re getting it from?
Okay. That’s working.
It’s working? Yeah, that was taking some time. The real Heather Mickman certificate is here, just for you to gaze upon. I just love this. Yeah, this is actually based on a real thing. Levi Geinert and Dan Cundiff talked about the next generation architecture and that was actually done… This is part of what is talked about in their multi-day offsite with the CIO, Mike McNamara. I think their goal is to basically categorize technologies into three categories. We’re using it recommended, there’s a strong community that’s using this technology and we have tons of experience in it and we love it.
Neutral, meaning the jury’s still out and eliminate meaning we have made a decision as a company that we’re getting off of this and so Scott Prugh’s presentation 2019 was all about this. Part of their goals is to eliminate their reliance on hostile vendors as measured by, “These are the people who increase our prices during contract term renegotiation. They’re like, “These people are actually jeopardizing our business.” I love that. I would recommend you watch the 2019, I’m sorry, 2018 presentation from Target from Levi Geinert and Dan Cundiff. It should give you some ideas, and also speaking of which there’s actually another presentation I would recommend, which is from HP.
This is before the breakup by their CIO, Ralph Loura, I think it is in 2015 and they talk about how the goal was to create boundaries, not buoys. Buoys, not boundaries. Yeah. The goal was to say you one of the safe channels in the Harbor, but if you need to stray outside the safe channel and you can follow the principles you’re authorized to do so and maybe that will become adopted into one of the next points of greatness within the organization. It’s a phenomenal presentation. Yeah, I think those hint at what next generation architecture review boards should be. In fact, I think that is Maxine’s job, it’s to create the next generation architecture for Parts Unlimited, next generation architecture review something.
Now I’m curious. What is her job description? Her job description is, oversee the architecture design… No. Guides creation of a governance and architecture review function that can evolve and ensure that the company obligations are fulfilled for years to come. Yeah. That is very much in the spirit of what Maxine is tasked to do. Seven minutes to go.
Question number ttwenty-one, Fernando, “Hey Gene! As you are an advocate for functional programming and it is clear how it is better in most scenarios, my question is: are there any exceptions? Meaning, have you seen situations where you’d recommend solving with imperative programming instead, because it would either be faster or take less memory than with functional programming, and if so, which one?”
I don’t know. Yeah. I’m sure you’re right. I think there are situations where I think in the Haskell community, they talk about, there’s some scenarios because of lazy evaluation, it’s really difficult when things go wrong to find out where it went wrong. Clojure has similar. It too is a lazy language that differs evaluation to when it’s actually used. I’ve heard performance sometimes use as a reason. Clojure, one of the problems is slow startup times associated with the JVM, but now with the Crawl VM thing you can actually compile it down to native code so it’s as fast as go, and I think even Rust. The Rust programming language is meant to essentially replace C++ with more functional programming and immutable sensibilities.
I think just the languages are just so much better now. It’s just amazing how much safer they are. I think there is some evidence to suggest that static typing is probably a good thing when you’re working with larger teams and that’s something that Clojure as a dynamically type language is not. It’s very much a fundamentally dynamic language like Python. There have been some efforts to do… Does something called spec, Clojure spec that allows you to do runtime checking, which I use all the time. I think when you talk about … In fact, in that Clojure Love Letter, there’s a video link to a John Carmack video. He created Doom, Quake.
Anyway, I love TypeScript and that’s not a functional programming language. Okay. I think we have one more. We’ll push for one more, Alex. Great question. It’s so much fun.
Question number twenty-two, Proctor, “Where does one find an Eric?”
Gosh, this one deserves a longer answer. Okay, let me touch on this one and then I want you to cue this up for next time, Alex. “Where does one find an Eric? I’m wondering how does one get progress as good as this level and it doesn’t have the air cover backing at the executive level like Maxine and Bill had with Eric covering them with Steve and Dick.” I don’t think Eric was an active champion in the way that you’re alluding to. It was really Bill. Eric didn’t care enough. Eric didn’t have to take any risk.
Eric wasn’t passionately advocating; he just knew what’s a better way it looked like. I don’t think Eric is what we’re looking for. What we’re looking for is a business sponsor and a technology sponsor. Just for better ways of working. We want those two people who are willing to go up at DevOps Enterprise Summit and brag about how the biggest problem facing organization was solved only through this incredible partnership with the technology leadership and business leadership. I think that sounds glib, but I actually think this is so true, is that books are such a powerful way to find these people.
As I put into the Slack channel, when someone says, I love The Unicorn Project or The Phoenix Project, it meant they spent six to eight hours reading it and it qualifies someone that says they recognize the problem exists, they believe the problem is significant and chances are they think it’s relevant to the work that they do every day. I’ve seen so many instances where books are able to serve as ways to find fellow rebels, find senior champions, and The Unicorn Project, I think in my ideal, it is a great way to find business people who care about it. In fact, there’s a LinkedIn post of Ken Kennedy who is Scott Prugh’s boss at CSG.
He’s the president of CSG, the person ultimately responsible for the business functions for a public company talking about how much he enjoyed reading The Unicorn Project and how greatness is created by through amazing DevOps principles and to a dedicated, amazing group of talented people. This is amazing. Here’s an example of me benefiting from someone reading The Unicorn Project to find someone who cares about this that I didn’t know about this before. I’m hoping that Scott Prugh will present with Ken Kennedy, his boss, the president of CSG, about how it is helping them achieve their goals, dreams and aspirations to win in the marketplace.
That would represent a level of seniority that we’ve never had at the DevOps Enterprise Summit before. I guess the reason why I bring this up is I am using this, The Unicorn Project in the same way that I think you will be using books to find your Eric, your Bill, your Maggie. You’re looking for the Maggie who is going to bring the toughest challenge and say, “I need your help to help make this happen because the current ways of working don’t work.” All right. Alex, with that, I’m going to turn it over to you and I’ll just comment. This was as stimulating, as interesting, as fun and as tiring as last time but still worthwhile.
We’re going to get this transcript created and we’ll make the videos available and I’m looking forward to the next one. Thank you for all your questions and your enthusiasm and I hope this has been valuable to you and I’m going to turn it over to you Alex. Thank you. We’re going to reconvene in two weeks, is that right, Alex?
Yes. We got the last one with Gene scheduled and thank you everyone for coming today and posting your questions. Follow the ones that we didn’t get to, I’m capturing and putting in a stack for next time. We’ll keep working through these. Other than that, let’s keep the week two discussions going. Next week will be the start of week three but right now we are so on it. I’ll get into Slack, answer some people’s questions, start some discussions and let’s keep going.
All right. One more last thing is that I’m actually working with a consultant, Alex and I were trying to get all of the Slack messages archived so they can be searched on Google and so forth and that’s done in Clojure. That’s been great fun for me. That’s something that we do to help serve this community and make sure that we don’t lose these. All right, man, thank you much. Thank you, everybody, catch you in two weeks and on the Slack channel.