Building an Automated Governance Architecture – Investments Unlimited Series: Chapter 5

Welcome to the fifth installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

Last time, Bill caught a pivotal glimpse into the future, foreseeing security “baked in” as code. Now, engineer Michelle corrals a ragtag squad to manifest this vision before the compliance clock expires! But can they dispel old grudges?

Tuesday, April 19th

Bill sat at his desk, staring out the window. It had been a long couple weeks since he stood in Susan’s office looking at the “DevOps has failed you” slide with Jason. Since then, he’d looked up the resources Jason had scribbled for him on that scrap of paper. They had turned out to be a couple of guidance papers. The first was short and simple, but it was powerful.

Dear Auditor was a simple letter written by an imaginary engineering team to their auditor colleagues. Reading it had made Bill realize just how much blame he and many others at IUI always placed on the Audit team. They seemed like the easy scapegoats they could throw all their woes at. But it had become clear that IUI was never going to drag itself out of this mess without seeing Audit in a new light.

He had immediately forwarded the letter to Michelle, who shared it with her Engineering team. They were hard at work trying to build a solution to their predicament. The team nicknamed themselves “the Kraken”—a play on Liam Neeson’s famous line “Release the kraken!” from the 2010 movie Clash of the Titans. 

Along the way, Michelle and her team realized they had to tackle their own perceptions and mindsets. They had even drafted their own version of the Dear Auditor letter and shared it with the whole company. They felt like this letter was a great way to set the tone for a fundamental change in how IUI’s Engineering and Audit groups operated, working together as a team instead of adversaries. They knew it wouldn’t change anything overnight, but they hoped it was a beginning.

Dear Auditor,

We realize that we have been rapidly changing our practices from Agile and DevOps to cloud and containers. Yes, we have been busy, and we are having great success delivering faster than ever with better quality, responding competitively to market pressures. However, this approach isn’t just icing on the cake. The only sustainable advantage in our industry is the ability to meet customer demands faster and more reliably than our competitors.

But with all this growth, we made a tragic mistake: we forgot to bring you along for the ride. That is totally our fault, and we want to make it right. We are going to make some new commitments, including the following:

• We will bring you along.
• We will be fully transparent with you about our development process.
• We realize that we own the risks of our business, and we will act accordingly.
• We will maintain an open channel of discussion to demonstrate to you how we manage risks with our modern development practices.

For example, you have told us that you are concerned about the separation of duties in Agile and DevOps practices, and we heard you! We have a better way to manage this issue now. Maintaining version control for everything we work on, enforcing peer review for all changes, releasing via a secure pipeline, restricting production access, and monitoring unauthorized changes in production systems are being enacted to help address your concern.

The DevOps community has been experimenting quite a bit over the last several years, and common practice now represents the collective wisdom across many companies, industries, and countries.

We have compiled a list of Audit concerns and documented them in a Risk Control Matrix with details about the controls, our practices, and evidence that has been collected to support each control. We hope this matrix provides a way for us to collaborate on risk mitigation practices from now on.

We stand by our commitment to providing value at a fast pace. We are regrouping in order to improve our processes with you, and we are truly excited to move forward together.

—The Kraken Team

Bill spun in his chair to look at the letter, which he had printed out and pinned to his wall along with a calendar ticking down the days until they had to respond to the MRIA. Alongside it was a picture taken at the last company-wide forum. The picture was no more than a sea of faces, but it, along with the letter and calendar, acted as his motivation. He and the team had to fix their governance issues before the deadline set by the MRIA. If they didn’t, everyone in that sea of faces would be unemployed. Not to mention the stakeholders they’d be letting down. Bill wasn’t sure how he’d be able to handle facing that kind of failure.

“It seems to be getting a lot of traction,” Michelle said from the doorway.

Bill, startled slightly, spun around to look at her. She was pointing at the letter he had been staring at.

“Yes, but it’s not a solution that we can pick up and run with. We still have a long way to go.”

Even though she had a slight smile on her face, Bill had been working closely with Michelle for long enough now that he could see there was something troubling her. It had been a hard few weeks discovering the extent of the mess they were in. He wasn’t sure if they’d be able to unravel the complex knot of requirements and systems, but he hoped the second resource Jason had given him would help the team unlock something.

“Everyone’s gathered together. You ready?” Michelle asked.

“Ready.” Bill grabbed his tablet and followed Michelle down the hall to one of the larger conference rooms. It was packed with the Engineering team tasked with helping Michelle. The seriousness of the MRIA meant they were able to assemble a special task force to focus on just this problem. At least, that was the intention. He’d be beyond surprised if they were able to make it to the end without someone getting pulled off to help with some crisis or another.

Michelle and the team were an eclectic mix of engineers. Some had bounced around different companies in the area. Others, like Michelle, were starting to make a career for themselves at IUI.

One notable team member was Omar. He wasn’t the most senior engineer on the team, but he was, like Michelle, a natural leader. Others tended to rally around him when they needed to solve difficult technical problems.

Omar had joined IUI three years ago and was not shy about sharing his opinion. He often jumped at a problem as though it was easily solvable only to realize later that it was far more complex than he had imagined. But when he saw the complexity, he just dug into it deeper and wouldn’t stop until it was fixed. Michelle welcomed his fearless attitude for this effort.

In addition to the Engineering team, Jada and Tim had requested people in their organizations to be part of the design process: Andrea from Audit and Barry from Security.

Andrea had started at IUI about six years ago. She never intended to have a career as an internal auditor; it had just played out that way. Andrea had worked her way up from a bank clerk to her current position. She was one of the few people at IUI who had taken advantage of the cross-training opportunities they offered. She found enjoyment in the audit process. She liked her house neat and tidy, and auditing made her feel like she was keeping the business neat and tidy too. 

Barry, on the other hand, was curmudgeonly and stubborn. A director under Tim, he was selected for two reasons. First, for all of Barry’s gruff demeanor, he was the most forward-thinking security professional IUI had. Second, Barry needed to be stretched. His management had noticed lately that he was starting to just coast. Barry’s manager wanted to give him something more challenging. They couldn’t tell if Barry was being lazy or if their asks were making him bored.

“Has everyone had a chance to read the DevOps Automated Governance Reference Architecture document that Michelle and I shared with you?”

“Yes, it was duller than a butter knife,” Barry said curtly.

“Really? I felt like the authors knew our release management approach intimately,” Omar observed. “They wrote about how to make it better in many ways I’d never thought about before.”

Andrea interjected. “The framework they used to identify the inputs, outputs, risk, controls, actors, and actions was very clear. But I don’t understand things like build or dependency management. Though, even without that context, that paper laid out a good approach to ensuring that whatever happens in those process boxes, the outcome is clearly auditable.”

“I agree,” Bill said. “What I found most interesting was how this automated process could help solve our problem of delivery velocity. I think we can all agree that as IUI has grown, our rate of delivery has increased quite substantially. It has become very difficult for us to stay in compliance with our own security and compliance requirements without slowing down.”

“Exactly,” Michelle added. “It’s not that the developers want to do the wrong thing. They’re not actively trying to break the system.”

“You could have fooled me,” Barry replied.

“But when it’s so hard to stay in compliance,” she continued, “when the road is full of potholes and obstacles, well, we’ve had to find ways around those so that we could still meet our delivery requirements.”

“I had a similar discussion not too long ago with Jason. Jason was explaining that our incentives are causing these problems. Developers are incentivized to go faster, delivering more features quickly, and I’ve been supporting that. Ops, Security, and Audit,” Bill said, nodding at Barry and Andrea, “well, you all have been incentivized to reduce risk, of course. Which in our old system slowed things down so that Dev couldn’t ship features quickly. It’s a conflict. I think Jason called it a core chronic conflict.”

“Talk about a merry-go-round of doom,” Michelle said, chuckling. Andrea smiled back at her, but Barry didn’t look at all amused. “Uh, let’s get back to this paper. The DevOps journey we’ve been on has helped Dev automate more and more practices, moving faster and faster. And, well, like the ‘Dear Auditor’ letter showed, we’ve left you all behind. But it doesn’t have to stay that way. This paper shows we can automate governance! I mean, how wild and exciting is that!”

“Totally!” Omar agreed. “This paper shows how we can implement an automated process to track all the governance requirements throughout the delivery pipeline. It’s wild!”

“I think this could really be the answer to our problem,” Michelle responded. “I mean, we have no choice other than automating our software delivery governance anyway. We’re not going back in time.”

“This is great enthusiasm,” Bill said. “But I want to make sure we are baselined on the problem we need to solve. In the MRIA Outline document Michelle and I have shared with you all, we are focused on designing an approach for the actionable items. It’s important to remember that the approach we choose needs to provide a paved road. That is, an easy, or at least easier, path for Dev to follow.”

“Yeah, otherwise developers are going to feel like they’re just getting hit with a boatload of new policies that will make meeting their incentive of fast delivery even harder. And then they’ll continue to find more workarounds, and we’ll still be out of compliance. We’ll still be in the same mess we’re in right now.” Michelle took a deep breath. She looked back down at the printed MRIA Outline doc. “So, the actionable items are:

Actionable Items: Based upon the MRAs issued, the following items should be addressed with formal standardized approaches:

• Goal: Define a minimally acceptable release approach
• Objectives:
  • Enforce peer reviews of code that is pushed to a production environment.
  • Identify and enforce minimum quality gates.
  • Remove all elevated access to all production environments for everyone.

“I see why we’re in this mess. You all haven’t the slightest clue how to keep your house in order,” Barry blurted out.

“Who left surprises in your cereal this morning?” Omar responded. He wasn’t having any of Barry’s attitude.

“Michelle, are you saying we need to replace our pipelines?” Bill inquired, ignoring Omar.

“No, we’re not trying to replace the delivery pipeline. To meet the MRIA, we need to have a documented approach. That’s first. Second, we want to bring in this automated governance approach, which is about augmenting existing pipelines and tool chains,” Michelle explained. “The original DevOps ideals addressed the disconnect between development and operations. It didn’t consider the roles of Security, Compliance, Risk, and Audit. So we’re just trying to bring you all into the fold, all without having to slow down.”

She picked up her tablet and continued. “Turn to page eight of DevOps Automated Governance Reference Architecture. I think this could be the baseline for our documented minimally acceptable release approach.”

Michelle projected her tablet onto the screen at the front of the conference room.

“That has nothing to do with security,” Barry commented.

“Barry, it’s a baseline,” Michelle responded. “We need to build upon it. Do you have recommendations for injecting security considerations into this?”

Barry paused for a moment. “I wasn’t brought here for this.”

“Yes, you were,” interjected Bill. “The design of our new approach includes security and auditing concerns. Our goal is to shift these aspects of the software delivery process left. You’re an expert regarding security concerns. Security in our process is not an option. Your involvement may be. Should you and I speak with Tim about finding someone more willing to help?”

The room fell silent. Barry sat there, stone-faced and emotionless, not even appearing to think about what Bill just said.

“Okay, I apologize. That’s not necessary,” Barry started. He looked back at the screen, obviously in thought. He was silent for a few minutes before adding, “I may have a couple ideas. Let’s continue and I’ll get back to you all.”

Bill smiled, sensing he’d won the battle. “Okay, looking forward to your input.”

Michelle, pleased to have some of the tension eased, pointed to the screen. “So this is our generic pipeline. Our teams develop code, automatically build the target executable while pulling down various dependencies from the dependency management system, package the necessary files together, and publish the package into our artifact repository. Next step in the pipeline is to deploy the package to non-production environments  .  .  . ”

Barry interjected, “Do we not test what we write?”

Michelle answered, “As I was saying, this is a baseline  .  .  .  of course we test, and that’s why we deploy to a non-production environment first. We test there, and if everything looks okay, we finally deploy to production. We have some common controls across the pipeline stages and the tools that we use. We also have controls around who gets access to our tools and how they access each tool’s features. We have audit logs  .  .  .  are you all following me?” Michelle looked around. 

Everyone was looking at her and waiting. 

“Okay good.” She smiled and continued. “Our digital team has every pipeline code in our source control tool. So next, we need to talk about how we show evidence of our actionable items.”

Michelle projected a new image onto the screen. “Here is a simple table outlining a basic set of attestations. And by ‘attestations,’ I mean our claims of how we’re meeting these controls.”

“Bill and I went down the rabbit hole a couple times trying to think of everything. Then we realized we were just spinning our wheels. These seem like a good place to start.” 

“Have you considered where you’ll get the evidence and how we’ll collect it? And how do we apply policies?” Andrea inquired.

“Yes, that was part of our Wonderland excursion during discovery these past few weeks. That’s why this team is here. Let’s structure an approach to answer the same questions you asked, Andrea,” Michelle responded.

“What about the tools or systems to automate this?” Omar asked next.

“We talked about that, but not much. If we start with a tool-first approach, we may miss the forest for the trees. I think it’s best if we design the business process of automated governance first and then do a tool and technologies selection.”

“What?! We can accomplish much more if we just pick some tools and go,” Omar challenged.

“No offense, Omar, but that comment right there is how we got into this mess. If we couple ourselves to the implementation—if we just do what the tools can do—than we will always be limited to the capabilities of our tools. What happens if we need a capability that the tool doesn’t offer?” Michelle replied.

“We build a tool and integrate it,” Omar responded quickly, as if it were the only truth.

“Omar, I think you’re missing the point. I can get with you later to talk about architecture versus implementation. Implementation first is akin to the tail wagging the dog,” Michelle said as a means to end the distracting conversation.

“Michelle, you’re talking about starting with business architecture, correct?” Andrea asked.

“Yes! No offense, but I’m surprised you’ve heard about that,” Michelle responded.

“Don’t mistake me for anyone who knows more than those two words,” Andrea quipped. “One of IT’s enterprise architects had training on some open group framework thing. All I remember was their incessant repetition of ‘business architecture first.’ When I look at this reference paper, it makes me think of this as our business architecture, our business process.”

Michelle looked delightfully flabbergasted. Seeing this as an opening to practice the Socratic method, she asked, “How do you think we apply this to our business architecture?”

“This DevOps automated governance will be our business process. We don’t have to create anything. I’ll take it one step further than just a baseline; let’s start with this. Our goal can then be to move from left to right. Borrowing that Agile stuff you all talk about with iterations. Look at page fourteen,” Andrea said, pausing.

Michelle took her tablet and switched from page eight to page fourteen on the big screen.

Andrea started back with, “Stage One, Source Code Repository. The risk says Unapproved Changes and the control says Peer Review. Isn’t that the same thing as our first actionable item of Enforce peer reviews of code that is pushed to a production environment?”

“Yup,” Omar replied. “If you look at page seventeen, it has things like unit testing, static security analysis, linting, and some other things I’m not familiar with  .  .  .  like immutable builds. Those look like a good start for one of those action items.” Omar rustled through the papers in front of him and found the MRIA Outline printout.

“It’s item two, Identify and enforce minimum quality gates,” Omar read aloud.

Bill said, “Michelle, either our tail-chasing yesterday prepared us better for today’s conversation, or Andrea, Omar, and Barry are better at getting to the point than we are.”

Bill’s attempt at flattery wasn’t making Barry any more cheery. Bill continued. “We have a way forward for our first item, some ideas around the second item, so now we need to address the third: the developer access.” He looked back down at this printout. “Remove all elevated access to all production environments for everyone.”

Bill didn’t have to turn to Barry to signal for Barry to speak up. “Every new application has some type of break glass request that comes my way,” Barry said. “We’re always letting them into production machines, and most of these are not really break glass—they’re not tracked and they are probably not revoked. I loved containers until the developers realized they could push into them. Omar, you should learn what ‘immutable’ means.”

A mixed expression of embarrassment and interest came over Omar’s face. He kept silent. Even though Barry’s delivery was as rough as sandpaper, Omar knew all too well it was the truth.

Barry continued, obviously trying to be helpful so he didn’t get an earful from Bill again. “If there is one thing this old man does, it’s keep up to date with what’s going on. I was at a DevOps Days event where someone was talking about tracking this. They had a metric for it; they called it ‘Production Access Debt.’ Every time a persistent production account is accessed, you add ten points. Each break glass read account is one point and each break glass write access is five points. The idea is to keep a zero value as an elite attribute. It’s intended to drive the correct outcomes of automating and no human server access. This metric drives the correct outcomes, like more automation and less access to a server.”

“That’s very interesting,” Bill said, encouragingly. “Can you tell us more about that?”

Barry got up from his seat and walked to the dry erase board. Grabbing a marker, he said, “Here’s what I think we could do for this action item. Again, this is based on some things I’ve seen at recent conferences.” He started making a list.

• Everything must be code.
• All logs must be streamed out.
• No system in production unless it has observability built in.

Looking at the board, Omar asked, “What do these mean? I mean, why?”

Barry smiled. “You know, I’ve been listening to you all. And I’ve seen what developers are doing around here. Everyone is cheering about automation, end-to-end CI/CD pipeline, immutable infrastructure, SRE, observability, and all these shiny new things. But I want to ask our developers one simple question  .  .  .  if we really are that good at DevOps, automation, and infrastructure as code, why would anyone still need production access?”

Barry paused, looking for reactions. Omar was looking at the floor.

Michelle spoke up. “You’re so right, Barry! I completely agree with you. I already discussed the ‘everything must be code’ concept with Tim and Jada, and they understand where we’re going with this.”

Barry grinned. “Really! I wasn’t invited to that meeting. Guess I’m not that important.”

“Oh, come on, Barry! You know it’s not like that,” Michelle said. “Tell me more about the other items on your list.”

“Well, you want to remove all elevated access to production,” Barry explained. “But taking it one step further, I think we should remove all access to production.”

“Can we do that?” Michelle asked. “If we can, our auditors will love us! They probably won’t believe us either, but that’s another story. Right, Andrea?”

“I guess,” Andrea laughed. “If you show me the list of the names of people with production access and the list is blank, I’ll believe you!”

“To do this, as you all know, you need all code, including infrastructure code and of course the pipeline code, in source control,” Barry continued.

“What’s pipeline code?” Bill asked.

“It’s the same idea as infrastructure as code. You do know what infrastructure as code is, right?”

“Yes, that’s about the only thing in infrastructure I know about,” he replied.

“I’ll send you a link to a blog post from Capital One called ‘Governance in a DevOps Environment.’ In it they wrote about taking concepts from infrastructure as code and applying it to CI/CD pipelines. The pipelines themselves are source controlled, just like the application code that passes through the pipeline. That way, changes to a pipeline are visible and provide a way to prevent unauthorized changes to how a pipeline operates. Not keeping an eye on the pipeline itself is how you find yourself in a situation similar to what happened to SolarWinds.”

“Tell me again why you want to remove all production access?” Omar asked.

Barry answered. “Because first, we want to make sure no one manually deploys code to production or makes any kind of changes there, and second  .  .  . ”

Michelle jumped in with excitement, “And second, we all can encourage modern practices around logging and monitoring. In fact, with specialized log analysis and observability tools, we get a direct line of sight into the health of the system. We can even detect issues before they happen.” Michelle paused to let that soak in before adding, “Our complex systems are noisy. Having a clear signal will reduce the cognitive load on our teams. Reducing that stress frees up capacity for higher value work. ”

Barry added, “And that’s exactly the reason why I have the other two points on my list.”

There was silence in the room. Everyone thought this was a good idea, but it sounded too scary and involved too much work.

“So, as I was saying, if we do this, I can happily revoke all the developers’ access. It will be a sweet win. I may even do it over a bottle of Scotch,” Barry finally cracked a smile, surprising everyone.

Michelle found Barry’s insights amazing. His previous life as an infrastructure admin was showing through. 

The engineers on Omar’s team had a ton of questions for Barry. Bill, Michelle, and Andrea sat by, listening intently. Both Bill and Michelle knew this type of open-ended conversation needed to happen. It’s a common way to flesh out a design.

Bill looked up at the clock and realized they were nearly out of time. An hour had passed in a flash. He called the room to attention. “Hey, all.” The room quieted down. “We need to wrap this up. Before we can move forward, I need a show of support for this proposed path. As I think Michelle has laid out, and this paper supports, automating governance is going to not only help us meet the concerns laid out in the MRIA but also, and maybe most importantly, prevent IUI from ever finding itself in this position again.” Michelle nodded enthusiastically. “So, do we have the support of your respective departments to use this approach?”

Bill looked at Barry and Andrea.

“Obviously we’ll have to take this up the chain,” Barry said. Andrea nodded in agreement. “But it does look intriguing.”

“Great. Okay, Michelle and I will formally document everything we did here today and send it out no later than tomorrow afternoon. We need you to take this back to your respective organizations and have conversations with your folks,” Bill said, turning to Andrea and Barry.

“Let’s get input and other insights as well. We’re partially looking for any ways to make this better, but primarily we want to use this opportunity to start ­communicating our proposed changes. Letting everyone know where we’re headed and where we may be headed is critical.” Bill paused for a second in thought, then continued. “Let’s take the next week to get feedback but also to experiment. We need to get to an implementation approach, but let’s make sure we don’t put the cart before the horse. Any questions?” Bill asked, taking another momentary pause.

Omar raised his hand, then started, “I have an idea for a tool.”

Michelle quickly cut him off. “Omar, let’s hold that for the next discussion. I’ll set up a new chat room in the IUI messenger so we can have an open conversation. Put your idea for the tool in there.”

Frustrated, Omar curtly replied, “Sure.”

The meeting was over. Everyone started to gather their stuff and head for the door when there was a muffled ping. Those left in the room started patting their pockets for their phones.

Pulling his phone out, Bill saw a text message from his wife—a reminder to pick up a good Cabernet on his way home. She was cooking Wagyu for their dinner that evening.

“Do you feel confident you have enough for Susan’s next huddle?” Michelle asked.

“Yes, yes I do,” responded Bill, looking up from his phone at Michelle. “Listen, there’s a BOGO sale at the supermarket and two bottles of wine with my name on them. Send me your notes so we can summarize all of this for the team, like we promised,” Bill concluded as he rushed out the door with dinner on his mind.

United by a “Dear Auditor” letter, the first rays of collaboration shine through! But Omar’s impatience threatens early harmony. Can the team align on an initial architecture for reform as pressure mounts? Join us next time for the continuation of the story. Or, go to your favorite book retailer and pick up a copy of Investments Unlimited today.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•