DevOps Failed You – Investments Unlimited Series: Chapter 4

Welcome to the fourth installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

In last week’s installment, rising star Michelle had to untangle the gaps in IUI’s governance. A nasty blame game erupted as Michelle unveiled alarming truths top brass don’t want to hear! Yet this week, inspiration strikes from an unlikely source. Will a fateful glimpse into CEO Susan’s office crystallize Bill’s vision?

Wednesday, April 6th (continued)

The room emptied fast, but Bill stayed behind. He sat in his chair, contemplating the position he’d just put himself in: right in front of the firing line. This wasn’t going to be easy. The thought of managing upward, sideways, and downward made him dizzy and uncertain. He pondered the next steps.

He needed to think. He packed up his stuff and tidied the chairs as he left the room, planning to have a beer over lunch at the local pub. The thought of an afternoon stout and some Cypress Point raw oysters made him happy.

On the way down the hall, he passed Susan’s office. As he walked briskly past her door, he could see someone presenting to her.

For a CEO in a highly regulated company, Susan was pretty open. She lived the company values by never shutting her door unless it was absolutely required. Many folks were polite enough not to bother her when someone else was in the office, but it was generally accepted that you could if you needed to.

Bill’s attention was grabbed by a slide that was projected on the wall monitor. Susan was reviewing it with a bespectacled man, Jason Colbert, the SVP of Digital Transformation. Jason had been brought in to lead the continued DevOps transformation of IUI a year ago, and they had just started working together on a new digital product strategy. 

Jason was waxing eloquently about his presentation. There wasn’t any formal style to this slide, just white words in bold, simple type on a charcoal background. The words read “DevOps failed you.” It grabbed him.

Bill couldn’t hear clearly what Jason was saying. He walked closer to the door and waited to see if he could make out anything. 

He gave it a couple more seconds, turned around, and then walked past the door again.

That slide was really intriguing to him. No one else from IT was in the room. And why was DevOps under fire? They were one of the few firms he knew of that had successfully transformed from an outdated “Plan, Build, Run” siloed way of working to a more collaborative, product-focused approach. IUI was among the first few companies that participated in DORA metrics. Why was the head of IUI’s digital transformation now suggesting that DevOps had failed?

Finally he caught a snippet of the conversation.

“DevOps failed you. But it wasn’t DevOps’ fault. Most people forget to apply systems thinking to their DevOps transformation. In fact, Jabe Bloom has some great blogs on this you should check out. He calls it the ‘Three Economies.’

“You’ve probably heard the term ‘DevSecOps’ thrown around these days,” Jason continued. “It’s not just another buzzword; it’s an admission that there is more to DevOps than just Dev and Ops. An organization must go beyond developers and operations. They must consider everyone in the value stream.”

Susan was looking at Jason very seriously. Bill could only imagine the stress she was under and the pressure she was getting from the board with this MRIA.

Before Bill realized he was staring, Susan looked directly at him and smiled.

“Hey, Bill, come on in.”

Bill jumped at the sound of her voice.

That dizzy feeling came back to him. His palms began to sweat. He was very uncomfortable knowing he’d been caught like a child hiding on the stairs peeking between the banisters and listening in on his parents’ conversation way past bedtime. He decided to style it out.

“Sorry for the office creeping,” he began. “I saw the words ‘DevOps failed you’ and I couldn’t help myself.”

Susan smiled broadly. Jason turned around to face Bill, a jovial grin on his face. This wasn’t what he expected. From the tone of the conversation he had overheard, it sounded like they were discussing a dire situation. But now Jason and Susan were both all smiles.

“Yes, Jason was just teaching me a thing or two about how to build better software,” Susan said. “Can you imagine that. A CEO learning about building software.”

“Hiya, Bill!” Jason said with enthusiasm. “What can I say? I’m a recovering academic and sometimes I get a chance to practice my old trade. Susan is kind enough to let me chew on her ear from time to time. We were just discussing the audit findings, and I was sharing some observations and insights.”

Jason always had an excited yet relaxed tone. Today he wore wrinkled khaki pants and an untucked denim button-down shirt. He seemed jolly and talked as if he’d just stuck his hand in the honey pot and was licking away. Definitely not what Bill thought of as a typical Boston academic.

“By the way,” Jason said to Bill, “we have a couple of Sicilian cannoli left over from Bova’s Bakery. Best cannoli west of the Atlantic, you know.” He nodded at the pastries, and Bill grabbed one gratefully. The oysters could wait.

“I was just telling Susan how DevOps ideals are great, but they’re rooted in the core chronic conflict between Development and Operations,” Jason said, turning back to Susan. “‘Core chronic conflict’ is a fancy way of saying that people across the organization are incentivized in ways that prevent cooperation, therefore preventing the achievement of organization-wide goals. This has directly led to the mess IUI is finding itself in today with the MRIA.”

“Our incentives are causing these problems? How?” Susan had lost her smile and was once again listening to Jason with a serious look on her face. It made Bill feel the weight that she obviously had on her shoulders. He took a bite of his cannoli and sat on the edge of her desk, listening closely as Jason explained.

“Developers are incentivized to go faster, delivering more features quickly.” 

Bill nodded while taking another bite of his cannoli.

“Operations,” Jason continued, “are incentivized to reduce risk of change. If you’re delivering features quickly, you’re changing at a fast pace. Therefore there is a core chronic conflict between Development and Operations. Moving quickly versus the risk of change.” 

“Yes, I think we were just coming to a similar finding during our MRIA meeting a few minutes ago,” Bill said.

Susan turned toward him, and he suddenly felt like he’d put himself on the spot. “We were looking at how Development might have inadvertently become a build trap,” he continued. “Paving at least part of the path to the mess we’re in now.”

“That’s certainly part of the problem,” Jason added before turning back to Susan. “When it comes to the audit findings, I think it comes around to the fact that the Security, Risk, and Compliance folks are trained and incentivized to think of every possible way a bad actor could compromise your system. Susan, I’ll send you a copy of the book Bill’s referring to.”

Susan nodded her appreciation.

“Josh Corman, a famous security specialist,” Jason continued, “says software’s not eating the world, as has been famously said; it’s infecting the world. Not that creating a lot of new software is a bad thing! It’s just that we’re constantly creating new opportunities for security compromises. Every time you add a new feature,” he said, looking pointedly at Bill, “these folks assess if the change created a compromise.

“It’s another core chronic conflict: developers are incentivized to regularly introduce features—the build trap you spoke of—and Security, Risk, and Compliance are incentivized to minimize the likelihood or impact of all known possibilities, which can take time if not done well, creating a problem for the developers’ need to move fast, and so it goes around and around  .  .  . ”

He trailed off, lost in a vivid recollection that had come to him. Finally he came back to earth. “Sometimes the actual language that the Security folks use can be blunt,” Jason joked. “I blame it on their vendors. So alarmist!” he chuckled.

Bill chewed on the revelation Jason shared. On the face of it, Bill felt this was obvious. He knew this was a big issue, but as far as he was concerned, it had always been this way, and he saw no prospect of it changing. Or could it? Hadn’t they already done just this sort of change with DevOps, like Jason said?

Susan looked at Jason. “So we need to think about how we DevOps-ify Security, Risk, and Compliance?”

“Exactamundo!” Jason said. “What IUI has been able to achieve with DevOps is great, but it’s not enough. And this MRIA brings to light exactly what we need to shift left next. I’ve been watching other organizations try to handle these types of issues. Some are treating security and compliance as if it’s a feature of their product.”

“Hold on, did you say ‘treat security and compliance as if it was a feature’?”

“Yup,” Jason nodded.

Bill was surprised, but hearing the words mirrored back to him pushed his creeping panic attack away. It was replaced with cartoon scenes of bluebirds singing, bunnies hopping, and flowers blooming. He felt a smile appear on his face.

“We just said the same thing in my meeting with Tim and Jada,” Bill replied. “In fact,” he said, turning toward Susan, “they’ll likely be filling you in on just that at the huddle tomorrow.”

“I guess great minds think alike,” Susan chuckled.

“I’m not sure if it’s great minds or more software common sense these days,” Jason stated. “Take The DevOps Handbook. It points out three key aspects of DevOps: flow, feedback, and continuous learning.”

“Yeah, the three ways,” said Bill.

“You betcha. Well, those same concepts can be applied to Security, Compliance, Risk, and any other stakeholder along a value stream. These days, I’d argue that Development versus Operations is mostly solved. Now it’s all about systematically looking at all other parties that ensure the quality of software and including them in our shift-left mentality.”

“‘Shift-left mentality’? You’ve said that a couple of times. Can you explain that more?” Susan asked.

“Sure. Shifting left is a technique for bringing software testing as far forward as possible in the software requirements and design process. Imagine a diagram of the steps in the process. Testing typically happens after several other things and sits on the right-hand side of the picture. By moving it earlier in the process, we shift it left. Have you heard your developers talk about test-driven design and development?” Jason asked, turning toward Bill for confirmation.

“Oh yeah, TDD is one of Michelle’s favorite soapboxes,” Bill recalled.

“Good—that’s an important soapbox. When you shift things left, they become a forethought in the software design and development process, not an afterthought. Think of Security, Compliance, and Risk as a form of testing,” Jason explained, looking at Susan and Bill in turn. “Shift-left testing makes people think about how the software is supposed to operate, and then they codify tests. Then, once the tests are created, the engineers build the product and automatically run the tests to see if the product passes the tests. Testing this way is like creating a blueprint.

“If you watch how cars are built, before any car is produced, a blueprint for all components is designed. That blueprint is the specification for how the car operates. Testing functions that way as well, and so should Security, Compliance, and Risk.”

“To be clear, you’re saying we need to start baking Security, Compliance, and Risk into the upfront software designs?” Bill asked.

“That’s right! Just like testing, shifting security left allows it to be codified and automatically validated as the software is built. If it’s done correctly, the only human touch the software experiences, besides an end-user, is the smart people defining the tests, the smart people codifying the security and compliance policies, and the smart people writing the software. No longer do we need to have smart people performing audits by manually checking screenshots to validate things are all good. 

“Actually, that reminds me of a story I heard from a colleague at another company that shall not be named,” he said, placing a finger over his lips. “She found out that one of their teams had been uploading the same screenshot as testing evidence for sixteen months!”

“Oh, please tell me that’s not happening here,” Susan said over Jason’s laughs.

Bill was profoundly affected by this insight. It all made so much sense to him, but the problem was implementation. How were they going to actually make these changes? Why had no one talked about this before?

“As a product owner, I can prioritize security and compliance as features of my product,” Bill said. “If I challenge the team to shift these actions left as a means of validating the software, maybe I can help ensure we reduce audit findings. But the findings were about the software release process. I can’t change how they build software, can I?”

“Bill has a point. What do you think, Jason?” Susan said.

“Their processes reflect how you incentivize them. If you prioritize not just features but also how the features are brought to market, you give the organization a reason and the power to change the way software development is done. Most leaders and managers in digital native companies get this. They get this so well that you won’t hear them talk about DevOps or anything like what we’re talking about. They’ll just tell you that’s how they’ve always worked.”

Jason stood up for a second and did a little stretch.

“Bill, why don’t you find out what Security and Compliance needs, then work with the engineers to codify these needs into your software as if it were a feature?”

“It sounds like a good plan in theory, but I still wonder why Jada’s team doesn’t do this?” Bill asked.

Susan responded with a serious face. “Bill, I don’t know everything about software development, but I do know that IUI is not a digital native, and therefore change has to be made. You’re the person with a P&L incentive to see the change. You’re the person with the opportunity to shepherd the change and help lead Jada’s organization through the change.”

“So, where do you think you should start?” Jason asked, looking at Bill the way his college professor used to stare at the class when she presented them with an impossible challenge.

“Well, let’s see  .  .  . ” Bill felt like he was standing at the edge of a cliff.

“We’ve been talking about treating this like a product,” Jason offered. “How would you bring a product to market when you have no objective evidence the market wants it but some qualitative evidence that it’s desired?”

Bill mused out loud, feeling slightly more comfortable. “I would create small, quick experiments, minimally viable products, to see what does and does not work,” Bill offered. Yeah, he knew how to do that. He and his team did it all the time.

Jason smiled. “I think you’ve got it. But here,” Jason said as he walked over to Susan’s desk. Bill was a little amazed as he watched Jason riffle through her drawers, pulling out a pen and pad of paper. “I recently read a couple of documents that I think might help you and the Engineering team on your journey.” He jotted down some notes.

Jason walked back over to Bill, handing him the piece of paper.

“Now, those cannoli were a good start, but I’m hungry,” Jason said, turning to Susan. “Where can we get something more substantial to eat?”

Bill had a conceptual breakthrough as he finally grasped how compliance and security must be “shifted left” and treated as built-in product features! But smooth sailing remains distant as Michelle spearheads a motley team against the MRIA clock. Join us next time for the continuation of the story. Or, go to your favorite book retailer and pick up a copy of Investments Unlimited today. 

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Director, Global SRE @ Disney | Speaker | Co-Author of Investments Unlimited

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•