Investments Unlimited (Excerpt Part 4)

“Holy hell!” Jada nearly screamed as she read through the printed version of the MRIA Outline document in front of her.

Michelle looked around the room. The expression on the team’s faces as they read varied from stunned, shocked, and disappointed to what Michelle could only describe as outright disgusted.

“Forty-two systems  .  .  . ” Tim said in disbelief. “This is the type of information Susan needs. Not all of this nitty-gritty,” he said, flipping through the appendix sections, “but this first page tells the truth that needs to be told. Even though it hurts looking at this.”

“Tim, to make the most of this meeting, let’s discuss the Actionable Items listed here. Information like this is key to the IUI response to the MRIA,” Jada said.

“Okay, great,” said Tim. “Let’s start with the goal: Define a minimally acceptable release approach.”

“We’ve made great strides toward this in our DevOps journey, but we still have some gaps, particularly in digital banking,” said Michelle, looking at Carol. “Developers tend to use workarounds in legacy back end processes. Our documentation is either nonexistent or unclear. Developers who know the process follow it. But those who don’t know the process blame problems on legacy systems and the older applications developers who were here before our DevOps journey.

“It’s further complicated by the subjective nature of how we create evidence. For example, all the evidence is implicitly defined in our CMDB system without any hard or objective evidence being stored anywhere. Tim’s team runs reports against that CMDB system.”

“Yes,” said Tim. “And for the most part we believe we’re in compliance. But we don’t actually validate the data.”

“Why not?” asked Jada. No answer was forthcoming.

“Okay, what’s next, Michelle?” asked Tim.

“We lack code reviews before deployments. It seems everyone thinks everyone else is doing this correctly, but not everyone is.” Michelle laughed, failing to conceal the irony. She then continued. “There are a number of ways to bypass this process.”

“Like what?” asked Jada.

“Oh, I remember this one,” Bill jumped in. “Wasn’t there someone on one of your teams, Michelle, who built an automated script to auto-review his own work using a service account?”

“Say what?” Jada said, surprise evident in her voice. “And then what happened?”

Michelle replied, “We took his service account away. He left IUI last year. He came from a start-up and didn’t actually like our way of working  .  .  .  forget it. Let’s continue.”

“It’s actually really frightening to see that we have wide-open gaps,” said Tim.

“One of the issues is that we don’t have a formal way to request a peer review. Basically, a person committing code has to track someone down and literally ask them to stop what they’re doing to help them with the review. That is, if anyone bothers to do it. It’s disruptive, slow, inconsistent, and unreliable. We don’t allow for proper structure or bandwidth to support a formalized process for code reviews,” said Michelle.

“But, if you ask me what makes me nervous  .  .  .  we have too many old open-source and commercial libraries and packages. We’ve been using these libraries for so long that nobody pays any attention to them. They’re likely full of vulnerabilities and in need of patches. Heck, we don’t even know if we’re using correctly licensed third-party software.”

“That is bad,” said Tim, frowning.

“But what’s even worse,” Michelle continued, “is that there’s no consistent evidence of the security controls that Audit can easily examine. Some teams manually enter findings into their development backlog for remediation and others pass around PDF files or spreadsheets to their leaders. It’s completely scattershot.”

“And there’s still more?” asked Jada, with a hollow laugh.

“Oh, yes,” said Tim, wide-eyed, looking first at Jada and then the rest of the team. “I have heard that we lack controls, or tollgates, within the release pipelines. When Engineering teams started implementing automated pipelines, I was told that they would put some basic control gates in there. But I guess it never happened.”

“Unfortunately, Tim is correct,” said Michelle, shaking her head. “Over time, teams have been creating workarounds and getting exceptions to bypass the tollgates. As we’ve grown, so has our number of pipelines. It’s made it really hard to manage what happens in each one. In fact, many delivery teams have access to the CI/CD tools but have actually turned these controls off.”

“What?” said Jada, looking as if she had just witnessed someone running a red light in front of her. “How did that happen?!”

“Well, it looks like it started with just a few exceptions when there was pressure to deliver at a certain time and we couldn’t afford to delay the release. In time, though, it appears our system has become the ‘normalization of deviance,’” Michelle said, physically making the quote marks with her hands as she said it.

“Huh? The what of what?” someone in the back of the room asked.

“Diane Vaughan wrote a book called The Challenger Launch Decision. The whole Challenger disaster happened because deviance from correct or proper behavior became normalized in the US space program. It’s actually very common in corporate culture, and, well  .  .  . ” Michelle suddenly looked a little sheepish, “it’s happening here.”

“Great,” Jada sighed.

“And finally  .  .  . ” said Michelle, adjusting her glasses.

“Thank goodness!” said Jada, rolling her eyes.

“We have a very serious finding related to elevated access.”

“Like the other findings weren’t all that serious,” Tim smirked. A laugh rippled around the room.

Michelle calmly carried on. “The Development and Operations teams have a way to bypass the process. Basically, there are too many incidents of ‘break-glass’ access to systems. No one follows the guidelines. Team leads and managers grant approval left and right. I don’t think that the break-glass system even works.”

“But we have clear and published guidelines for this,” said Tim.

“I know,” said Michelle. “But people just ignore them. We don’t have a way to track elevated access requests. Our system has been open to abuse. And,” she added, taking a sip of her water, “it appears it has been abused.”

“This is worse than I thought,” Jada said. “We have to maintain segregation of duties. We’ll never receive a favorable examination with the regulators without it. Do I need to remind everyone in this room about Enron?!” Jada threw up her hands and looked around. There was silence. Michelle noticed a lot of people suddenly looking down into their laps. Throwing the name Enron around a bank was a surefire way to get people really uncomfortable really fast.

“Well,” Michelle piped up. “Actually, that should be achieved simply enough through the peer review system.”

“No way. We can’t have a developer pushing their own code into production and still achieve segregation of duties,” Jada clarified. “We’ll have to show that no one has both ‘developer’ and ‘operations’ roles. And the developer role cannot deploy to production. We also need to have a way to generate reports and compare the list of ‘developers’ against the list of ‘operations’ to ensure there is no overlap. We need to do that at least once every quarter.”

Michelle quickly interjected. “But Jada, we don’t have two roles anymore, not in the teams that have been ‘DevOps-ified.’ Now everyone is a developer. We’re either writing code for features or writing code for infrastructure. It’s all software now.”

“What?” Jada stood up, looking furious. “When did that happen? Who made that decision? Why was I not consulted?”

Tim leaned forward and commented sarcastically, “That was our DevOps transformation. The one we’ve been on for the last few years. Everybody said put Dev and Ops together, and that’s what we did. We made everyone a full-stack developer and put everyone in a single role: developer.”

Michelle sensed that the discussion was going downhill. She stood up and firmly said, “Listen, I was in a Lean Coffee session at a DevOps conference last year. The topic on my table was exactly this, segregation of duties. There were at least fifteen people at my table who were from other banks, retailers, software companies, and FinTech startups, even some folks from that famous auto parts manufacturer, Parts Unlimited. Everyone agreed unequivocally that segregation of duties is a joke. It doesn’t work.”

Michelle paused for a second to judge Jada’s and Tim’s reactions before continuing. “We assume that just because the code deployer belongs to another role, there will be no risk, or less risk. We’ve all seen the movie Office Space! Haven’t we?”

There were rumbles of acknowledgment from all corners of the room.

“Well, everyone at my table felt that segregation of duties presents an increased risk that someone not familiar with the change is actually putting the change in production!” Michelle looked at Jada. “And if you talk about a ‘bad actor’ scenario, an ops engineer can cause equal or greater damage than a developer.”

Jada quickly interjected. “So how do you ensure that no single developer has the ability to make a code change and deploy to production without anyone else’s knowledge? After all, that is the actual risk that we need to mitigate—and convince the regulators that we have it mitigated.”

“Exactly!” Michelle felt a lot better now. They were back on the same page, she hoped. “We have much better ways to mitigate that risk. We have all our application code in Git repositories. We use a CI/CD pipeline to build and deploy code all the way to production. Our pipeline is nothing but a set of codified build and deployment workflows, and we store them in the same Git repository, along with our infrastructure code. Now, if we take away elevated production access from every developer and ensure that every code change is peer reviewed before production deployment, we will have the best way to mitigate that risk that you mentioned, Jada. The key is enforcing the peer review process.”

“So,” Tim said, stopping Jada before she could respond. “Do we do any of that now?”

Michelle looked away and said, “No.” Lowering her voice and sounding a lot less energetic, she continued. “I mean, I know that most teams have pipelines and infrastructure code in their repositories. But that’s about it. And as I was saying earlier, our peer review process is broken too.”

Tim looked at Bill and said, “I think we all understand the extent and nature of the problem. Let’s talk about roles. Bill, as Product Manager, you have ultimate accountability for digital banking.”

Bill had sat through the discussion quietly, watching the room. As Michelle had gone through the findings, he’d observed Tim and Jada’s reactions. The situation that IUI had found itself in was finally starting to sink in, and he was feeling fearful.

“Yes, but am I the right owner for this? These aren’t features. These are security and compliance issues. Engineering problems.” There were murmurs of agreement from many in the room. A voice in Bill’s head wondered what his security and compliance colleagues did besides tell him what he was doing wrong. He glanced at Michelle’s face. It was evident that she didn’t intend to take ownership of these problems.

“But,” said Tim, “one could look at compliance and security features as non-
functional requirements in a product, right? If a security hole gets exploited or a compliance control is not met, that means the software has a bug or undocumented feature. And you don’t like those, do you, Bill?”

“No,” said Bill. “I really don’t. Michelle and I actually had this conversation just the other day. She was quoting a guy  .  .  .  James  .  .  . ”

“James Wickett,” said Michelle, smiling.

“That’s it. You heard him talk at a DevOps Enterprise Summit session on security bugs. He said: ‘A bug is a bug is a bug.’ That’s right, isn’t it, Michelle?”

“Yup,” Michelle nodded.

“You know, as Product Manager, it’s my job to ensure a product’s features meet market needs. And we do that. We deliver a lot. Quarter after quarter we’ve met and exceeded expectations. But with all that output, I’m wondering what actual outcomes we drove. I’m wondering if we created a build trap.”

“What’s that?” asked Tim.

“It’s something I’ve read about recently,” said Bill. “It’s a concept where if you only focus on delivering features and neglect experimentation and learning, than you likely aren’t building the right thing; moreover, you’re not learning or improving how you’re building, which is our case here. We never put stories in to improve and possibly re-architect our build process for complete traceability and compliance. This is all about shifting left on security, quality, and resiliency.”

Bill’s revelation seemed to land well in the room. Everyone looked at Bill, then at each other. They all started to realize how these MRAs could have been ignored for so long. Everyone was so busy trying to keep up with daily work that they had no time to improve daily work. That included security, quality, and resiliency.

Bill continued, “To do this the right way, we’re going to have to bring in several teams from the business. And you know those teams will likely refuse to change how they work. This isn’t their process. It’s not going to be easy to get this new idea through those silos. But I think I know how to have this discussion with stakeholders. They will sometimes need to wait a little longer for a product. We can’t maintain a ‘move fast and break things’ culture if it means we end up in a mess like we’re finding ourselves in today. But I still wonder, if this is a security and compliance problem, shouldn’t it be owned by Risk?” He looked at Jada.

“We’ll be right there with you,” said Jada. “But it’s your product. You should own the features. And, as Tim said and you agreed, this is a feature. We’re going to have to learn to collaborate better and create a pattern of working for the rest of the organization to learn from. I don’t think we’ve ever done this together before.”

Tim agreed. “Jada, we’ll need to include members from my Security teams and your Risk teams to participate in this project. I’m guessing Barry and Andrea are key people we should have working on this because  .  .  .  ?”

Jada nodded. “I’ll get us a meeting set up.”

“Okay, it sounds like we should treat this like a market problem,” Bill said, though he still felt a little uneasy about it. “We’ll need their expertise to understand what I’ve been missing when planning a product. And I’d like Michelle to help drive an engineering solution.”

“Sure,” Michelle responded. She had been surprised at how well she and Bill had worked together over the last week. And after listening to everyone’s pain points and problems, she was eager to begin crafting a solution.

“Okay, I think we all know what to do next!” Tim exclaimed. “I’ll take what we’ve decided here today to Susan at the huddle tomorrow. Then we’ll reconvene in a week to share what headway we’ve made. Let’s do this like we did today and have all the material ready before Susan’s weekly huddle. Michelle, I’ll set up a kick-off time with you.”

“Sounds great,” Michelle replied.

Read more in the new book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis. Coming to a book store near you September 13, 2022.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
•