The Three Lines Model: Investments Unlimited Series: Chapter 8

Welcome to the eighth installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

In our last installment, the team was making headway on shifting left with security. But now, a disastrous data center outage sidetracks our team just as vision crystallization beckoned! Persnickety higher-ups now clamor louder for outside saviors while Michelle fights to keep her exhausted, drained-dry Turbo Eureka afloat. Is an engineering mutiny next, with the deadline fast approaching?

Monday, June 6th

That Monday meeting didn’t happen.

Instead, Monday morning started with IUI’s customer-facing website going down. Customers couldn’t bank for nearly twelve hours. The Kraken team was pulled off Turbo Eureka and everyone was in firefighting mode, working well into the night trying to fix the problem.

Initially the blame was on IUI’s cloud hosting vendor. But the issue turned out to be a gremlin in a “lift-and-shift-to-cloud” project named Omega that IUI had hired an outside firm to assist with and which was completed months ago. The project had seemed successful at the time, but now it looked like it had caused more problems than it solved. IUI didn’t have the appetite to pull in enough funding to properly implement a cloud native solution. There was a push to “get on to cloud” by any means. At the time, the cloud infrastructure organization didn’t care about how it was done; they only cared about the number of applications that were migrated onto the cloud. Every week they published a chart that showed the upward growth of cloud usage. Senior leadership had been very happy about the progress they made.

The senior manager responsible for project Omega had no prior experience in cloud technologies. On top of that, it was his first big project at IUI. He wanted to make a name for himself and told the external firm to do whatever they needed to do to get the application migrated to the cloud. Project Omega was seen to be a huge success, and the senior manager got promoted. But a small number of IUI engineers who were close to that project knew that the external firm cut corners and broke all kinds of cloud engineering principles. They had raised these issues with their management team, but their complaints were ignored. It was only a matter of time before a real disaster would strike.

While the team was able to restore some form of service within a day, it was quite clear that IUI needed to immediately assign their best engineers to fix the contributing factors to the failure. Michelle and Omar’s team were pulled in to assist, along with many others. Michelle was worried they were going to fall behind on their delivery promises to the regulators.

Thursday, June 30th

Nearly a month into the incident, Michelle and her team found themselves at a town hall. Carol asked, “What could we have done better?”

Dillon spoke up. “For one, listen to us! We warned the team about this a long time ago. Every possible corner was cut to push this app into the cloud. Zero automation was put into place. They created layer after layer of single points of failure. Worse, without means for observability, there was no way for us to see problems. All of this piled up and grew even worse with each new upgrade. I guess you could say that technical debt finally broke the bank. But to be fair, we warned our management, and we know they warned leadership. This was avoidable, but clearly it wasn’t a priority until now.”

Gasps and muffled chuckles rippled through the crowd. Omar was doing his best to stifle his laughter. Michelle looked embarrassed not only for the Engineering teams but also on Dillon’s behalf. Dillon’s previous company had an open culture, where challenging things and speaking out directly was normal and welcomed. He was unaware of the mess he could be creating at this moment.

The town hall concluded quickly after Dillon’s comment.

“You know how you know when a town hall is successful?” Barry asked, leaning over to Andrea and Michelle. “When everyone arrives happy and leaves pissed off.”

Andrea pulled back from him and gave him a dirty look.

Walking down the hall and back to the dungeon, Michelle tapped Andrea on the shoulder. “Hey, how was it? You’re not normally in these meetings,” Michelle said.

“I had some time and decided to come and watch. Since I’ve been working so closely with your team, I admit I was curious. I feel bad for Carol; that had to be hard. Our town halls aren’t nearly as exciting as yours are,” Andrea replied.

“Not even when there are audit findings?” Michelle asked with a chuckle.

“No, nothing like that with audit findings,” Andrea replied. “Listen, we never met up to talk about my thoughts regarding the last demo. Now that things seem to be simmering back down, do you have time?”

Michelle responded, “I sure do. I’m free this afternoon. Now that it appears this incident is over, I need the team to refocus on Turbo Eureka full time. We’re about due for our next check-in with the external audit team and the regulators. Let’s meet at the high-tops in the dungeon in thirty minutes.”

Andrea and Barry were already in the dungeon when Michelle arrived. Barry was thumbing through a printout. A copy of the same printout was sitting in front of an empty spot at the table. Michelle assumed that was her spot.

Michelle picked it up. It wasn’t too thick, about ten pages of front-to-back print. The cover said Three Lines Model. 

“Michelle, thanks again for meeting with me. I hope you don’t mind, but I invited Barry as well. He and the Security team are key to this discussion,” Andrea said.

“Is this specifically about security?” asked Michelle.

“No, it’s about governance in general,” replied Andrea.

“Okay, I’m confused, although I think you may be about to explain this to me,” Michelle said with a smile as she took a seat.

“Yes, I am,” Andrea replied. “The Three Lines Model, formerly called the Three Lines of Defense, is a model for governance in highly regulated organizations like ours. It was created by the IIA, the Institute of Internal Auditors. In essence, governance is simply the process of identifying and making promises, and then checking that you keep those promises. The Three Lines Model attempts to bring structure to this concept. I was a bit concerned about Omar’s demo back before this Omega fiasco. What I saw there was a good start, but it didn’t seem to include aspects of the Three Lines Model, which we use at IUI to control risk. It goes beyond IT.”

“How do we currently do this in other areas? And what do the three lines look like when done properly?” Barry asked.

“Let’s start with our bank tellers,” Andrea said. “Bank tellers are the first line. The bank tellers at our organization own and manage the reduction of risk associated with their responsibilities. It’s very important to have their input into designing the controls because they are the people who execute the control procedures on a daily basis—they know which ones work and which ones don’t.”

“Are you telling me the bank tellers decide which controls to apply and which not to?” Barry blurted out.

“Well, in a perfect world, the first line would have input into which controls do and don’t work, but they are not the deciders of which controls to use. Actually, and unfortunately, most of the time they don’t get asked for input. The second line just railroads them,” Andrea responded.

“Audit railroading the working person. Sounds like we share a common understanding,” Barry said with some sarcasm.

“Actually, the second line is the risk management and compliance functions. Their purpose is really threefold. They decide how to structure the risk management framework for their respective organizations. They decide which policies and controls to put into place. And they’re responsible for making sure that the first line is executing within the established controls and policies.”

“So what does the third line do? It sounds like the first two do everything,” Michelle said.

“The third line is an assurance mechanism. They don’t decide what controls to use or how to implement them. They’re responsible for assessing if the risk management approach put in place by the second line is effective. The third line’s primary responsibility is to ensure that there are clear promises being made, that the promises are being kept, and that the whole promise approach is effective for the organization. These folks generally provide their findings to higher-level company executives and sometimes the board of directors,” Andrea said.

“Does this even apply to IT?” Barry asked.

“Here’s my recommendation, and tell me if this doesn’t make sense. Your first line is your engineers. They have the most context of your software, and they should be implementing the control procedures. The second line is Security. Barry, you should be working with the engineers to define and select the controls. The third line is Audit. If done well, it can be as independent as possible as a third party to see how someone like you, and say, Omar, establish a risk management approach and operationalize it.”

Andrea could see the wheels turning in both Barry and Michelle’s heads. They were thinking hard.

“What you just said isn’t really anything new,” Barry responded. “But—and this is a big but—how you laid out the roles and responsibilities of the three lines  .  .  .  well, I can appreciate that explanation. Most of our knucklehead engineers think I’m shoving security requirements down their throat. And most of the time, I am. But it’s because there’s no other choice. Ignorance pervades those keyboard monkeys.” Barry paused for a second to gather his thoughts. His eyes brightened as if an idea just came to him.

“Don’t let anyone know I said this,” Barry continued, “but the three lines perspective could make my job easier. Most of the time, I not only tell the engineers what controls to abide by, I also have to figure out how to implement them. With the three lines model, it would be up to the first line to figure out how to design and implement the controls. All I need to do is validate that their implementation is proper.”

“Can that actually work?” Michelle asked. “I mean, it sounds great in theory, but can it really work? Do we have the culture for it?”

Barry quickly responded with a full belly laugh. “No, not this culture. Although this culture is the culture that got us into this mess. So, evidently something needs to change.”

“I agree with you, Barry,” Andrea said. “Michelle, that’s why I wanted to talk with you. When I saw what Omar was doing, it was great, but it condensed all three lines into one. What he showed was as if the engineers determined everything. We need a way to ensure there are at least two lines.”

“Do you have any idea how we do that?” Michelle asked, looking at both Barry and Andrea. The pause felt like a lifetime. By now, each of them needed a brain break. There was significant contemplation going on and a great deal of angst to accompany it. Ideas were bounced around, but they kept wracking their brains on the what ifs. They were on the verge of damning an idea by what if-ing it to death before even trying it out.

Michelle started again, but this time she started drawing out a diagram. “So  .  .  .  let’s go back to first principles: What are the fundamental concepts? What do we need? First, we need a control. For example, peer review. Right?” She continued without waiting for an answer. “Second, we need a policy. Something that says a peer review must be done by one person other than the code author. There could be a legitimate case where more than one person needs to review a change. Which means we want to treat this policy as a configuration so that we can change it when we need to. Third, we need a way to collect evidence that a peer review was done. Fourth, we need to validate the evidence against the policy—in an automated way.”

“Isn’t Omar already doing this?” Barry asked, staring at the diagram Michelle was drawing. “I can’t believe I’m defending him right now, but his demo did all those things.”

“It did, but there’s a problem. He has a policy hard-coded in his tool. The second line, for example, should be able to change parameters of that policy in a way that doesn’t require the tool to be updated or rewritten,” Michelle responded.

“That makes sense. It’s the knucklehead knob. Externalizing the policy, like you’re talking about, allows us to turn things up or down while at the same time making sure these don’t cause any adverse effect on the developers. Is this the type of separation between line one and two you were referring to?” Barry asked.

“I think so. Some of what you and Michelle just said went a little over my head, but I think we’re on the right track. Can we prototype something like this?”

“The team would love nothing more than to get their hands dirty again with real, interesting work, not this ridiculous shift-and-lift remediation work. Barry, set up some time with Bill and Omar. Bill should be there so he can help prioritize features for this. And so he can understand the customer value. We can then rehash this conversation with Omar. I think we set them up with another spike and time box for two weeks. Based on the way they work, they’ll have something that meets our needs, although it may not be pretty. But we can worry about that later.”

Thursday, August 4th

Five weeks had passed since Andrea, Barry, and Michelle had discussed getting a new prototype built. The concept was simple enough, but some of the details were just out of reach or would require a bit more work.

Omar was always telling Bill and Michelle that he’d have a demo tomorrow. He did this every day the first week. Bill had nicknamed him the “little technical wolf” and joked that Omar would “huff and puff and tell you it’s easy, and then have more excuses than results tomorrow.”

Tension from the leadership team was also getting palpable. Susan had barely sat down for the latest huddle before ammunition began firing.

“It’s been nearly five months since we first got hit with the MRIA, and over two months since we began our execution plan. And there is barely anything to show,” Jada said in a loud, almost demeaning voice. “We need to show the regulators and external audit team that we’re making more progress.”

Tim joined her. “Barry filled me in on the lack of progress. I’m beginning to question if the engineering talent we have internally can pull this off. Carol, have you considered hiring an outside firm to engineer this for us?”

Carol was flabbergasted. Have they not realized we just spent a month cleaning up a disaster that project Omega left for us? she screamed in her head.

“No, there’s no need to assess another firm to do this for us,” Carol said. “There was a significant delay with the large system outage, as you all know. Tim, I believe it was a senior manager of yours who was responsible for that project.” 

Carol paused for a moment and looked piercingly at Tim. Tim had a scowl on his face but didn’t immediately say anything. Carol suspected that having Susan in the room was keeping him from saying exactly what he thought. 

“While I’m not intending to point fingers, when we fail, we fail as a team,” Carol said. “Unless you want me to make this about your organization’s capabilities?”

Feeling the tension, Bill said, “The team we have right now is the team responsible for some of our best products. We can pretend this is like everything else, or we can be realistic. This is an investment in continuous improvement. How do we prefer to couch it as leaders?

“Team Kraken has done something that had never been done before: they were able to automatically perform policy audits on their own internal software as it went through the build and deploy processes. Andrea and Michelle also spent a significant amount of time educating the engineering team on the IIA’s Three Lines Model. Collectively, they figured out how to reallocate human auditors from doing time-consuming and error-prone manual audits and instead focus on designing better controls and enhancing the IUI risk framework. Tim and Barry also started discussing ways to automate security controls,” Bill finished.

“I understand and can appreciate the progress,” Tim conceded. “But time isn’t on our side. We need to do whatever we can to get this done.”

Susan nodded her agreement. “Tim is right. The runway is running out. I know the team is working hard, but the massive site outage we just experienced has our board members even more on edge. We can’t afford to miss our deadlines with the regulators or it’s game over. This is our top priority. Let me know if you need any resources whatsoever.” 

Susan paused to survey the room and then added, “You have a blank check. I expect you to use it if you need it.” Susan stood up to leave.

The gravity of the situation hit everyone in the room and they all nodded in agreement, looked at each other, and stood to leave as well.

“Thank you, Susan,” Bill said and then looked at everyone standing and added, “Carol and I are headed down to check in on the team. You are all welcome to join us.” Jason enthusiastically agreed. Bill, Carol, and Jason left the huddle and headed to the elevator.

“You’ll have to tell me more about project Omega sometime,” Jason said as they entered the elevator.

Carol gave a small chuckle, and Bill shook his head. “That’s definitely a two-
finger Scotch conversation.” 

Jason laughed as the doors started to close.

“Dammit! You can’t be serious!  .  .  . ” was heard along with a string of other expletives as the elevator doors to the dungeon opened. Michelle, Bill, and Jason stepped off, awkwardly looking at each other like they had been dropped into a demilitarized zone. It seemed like the team was currently taking on heavy fire. “NO!” came another loud scream from across the room.

“Michelle, are we sure about not hiring an outside engineering firm?” Bill joked, trying to manage a smile.

“Bill, don’t test me,” replied Michelle.

They all rounded the hallway corner to see the engineers, Barry, and Andrea standing around Omar’s desk looking stressed and perplexed.

“What’s the problem?” Michelle asked as she approached the crowd.

“I’m not sure what’s happening. It’s not working,” Omar finally replied, nearly choking on each of his words. Omar spun around in his chair to look at Michelle. He quickly composed himself as he saw that Jason has joined the group. “We  .  .  .  well  .  .  .  we were looking good just a couple days ago. And the team thought we could add some of our other control gates to the automated governance approach in time for the next big demo.”

“No, not ‘we.’ You—you thought we could add other control gates,” Dillon interjected, staring intently at Omar. Omar shot Dillon a stern look, obviously upset.

“Let’s start from the beginning. Walk us through it all,” Michelle said, calming the situation down.

“I can’t. It doesn’t work. I need to figure this out first,” Omar replied.

“Omar, take a deep breath, step back, and let’s review it,” Michelle said.

“Yeah, review it, like with other people and get their input,” Dillon added.

“Okay, everyone. Let’s huddle up over there at the high-tops and work this problem together,” Michelle said. “Omar, bring your laptop over.”

Omar reluctantly gave in. He gathered his things from his desk, and they all moved to the high-tops on the floor. It was crowded—very crowded. From an outsider’s perspective, it might have appeared as if there was a sneak preview for a movie going on. There were even folks bringing over candy and drinks from the snack bar nearby.

“Dude, mouth closed!” Omar sniped at one engineer munching on a candy bar. “Okay,” he continued, taking a breath and trying to calm down after Michelle shot him a warning look. The usual collection of web browser tabs appeared on his screen. “We  .  .  . ” Omar started to say, but Michelle cut him off.

“Omar, before you start going through your code, walk us through the design,” Michelle said.

“Design?” Omar responded with a confused look.

“Yes, the design of the system. The components, how they fit together, and the interfaces,” Michelle responded.

Barry started to chuckle and said with a smirk, “Does your design include duct tape and bubble gum?”

Omar’s face started to flush red. 

“Omar, deep breath, seriously. It looks like you aren’t breathing,” Michelle said.

Omar sat back down, took his deep breath as instructed, and then looked back at Michelle.

“Remember the last demo, that report we created?” Omar asked.

“Yes,” responded Michelle.

“We took that file and used our policy engine to analyze it,” Omar said.

“What’s a policy engine?” Andrea asked.

“It evaluates our attestation against a policy,” Omar said. He continued on with the explanation. “Think of the attestation as a form, a form that is filled with information. For example, let’s talk about the code review attestation. The attestation has two fields: a field to list all the reviewers’ names who were not code authors and a field that has the count of reviewers. Let’s call these fields reviews and reviewers-­count. Then there’s another form, the policy. Each attestation has a policy, so for the code review attestation, there’s a code review policy. In that code review policy, there’s a line that says reviewers-count >= 1.”

Andrea spoke up, “Omar, does that mean there must be at least one reviewer?”

“Yup,” Omar replied. “You got it.”

“As another example,” Omar continued, “here’s an attestation with two reviewers, so the reviewers-count field equals two. Now it compares the numbers. Given two is greater than or equal to one, the policy engine returns a report that says the code review passed.”

“The term policy engine made me feel like this was more complicated than you just described,” Andrea said.

“It gets more complicated, but that’s the basics of how it works. We’ve been experimenting with a better way to assess code reviews for our Git repo—sorry, that’s a Git ‘repository’ for the non-developers  .  .  . ” 

Andrea gave him a thankful nod. 

“ .  .  .  with different types of policy engines. We played with OSCAP and OPA  .  .  .  uh, OSCAP stands for Open Security Content Automation Protocol. It uses a language called SCAP, or Security Content Automation Protocol, to define security controls and automatically assess and audit them,” Omar clarified, receiving a grateful grin from Andrea. “OSCAP is used in many places for assessing infrastructure compliance. I learned this while watching many online videos,” Omar said with a grin.

Michelle nodded and encouraged him to continue.

“There’s also OPA, or the Open Policy Agent. OPA is similar to OSCAP, except OPA is easier to use, or at least we think so. In OPA, you can write your policies in a language called REGO. We don’t know who is going to be writing policies for this yet, but REGO is a lot easier to use than SCAP. If we have to write the policies, we’d rather use REGO.

“Here’s a sample control we were playing with. We think it may be a better way to analyze code reviews than the way we do it now,” Omar said, pulling up one of his tools.

“I agree,” Andrea responded. “I want to use this as a prime example of how first- and second-line functions can collaborate in IT.”

“Omar,” Michelle called out, looking up from her own laptop. “I think I know your bug issue. Your JSON attestation file  .  .  .  is the object supposed to be named action or actions with an s?” she asked.

“It’s supposed to be  .  .  . ” Omar said, trailing off and looking sheepish. “You’ve got to be kidding me. It’s supposed to be with an s. Well, I guess this is a great example of the value of peer review.”

Highly embarrassed, Omar flicked through his screens, changing text in differing files. No more than two minutes later, Omar turned to Dillon and asked in a much softer voice, “Hey, man, can you approve the test pull request?”

With the click of a button, the web browser displaying the continuous integration tool interface came to life. The first blue bubble, which said Code Review Attestation, turned green. Then the bubble after that, which was gray, turned blue. Only a few seconds later, it turned green. This happened for the next four bubbles.

“That’s freaking embarrassing,” Omar said. “Sorry about that, everyone.”

“We’ve all been there, Omar. Sometimes we just need to slow down and take a breath, get some fresh eyes on something,” Michelle said.

Barry spoke up with uncharacteristic enthusiasm in his voice. “You know what we did today?” Looking around, no one said a thing. “We applied the same concepts of infrastructure as code to our governance. I’m going to go out on a limb with this one. Omar, what you showed with REGO, you showed policy as code. Our policies can be source controlled, just like our software and some of our infrastructure.”

“Policy as code?” Andrea asked, with a confused expression. “Does this mean that Audit and Risk need to hire developers and learn to write code? Your demo seemed great, but if we have to write code, I’m not sure this will work.”

“Um, I guess we didn’t think about that,” Omar said.

Michelle responded, “No, I don’t think so, Andrea. This is where we can collaborate. Based on how things are being built, someone will need to understand how to write the policies into the REGO, but it doesn’t have to be Risk. We can have a policy team. Andrea, or Barry, when we need to implement a control with this approach, an engineer can be there to help. They can write this policy file on your behalf. We can limit the people who can push to the policy-as-code repo, and we can even make you, or whoever is the right person in Risk, the approver for pull requests into that repo.”

“Is this how the new system will operate in the long term?” asked Andrea.

“I don’t know. What we could do is look at the next iteration. Possibly some type of user interface for the compliance groups?” Michelle said aloud.

Andrea had a hesitant look on her face.

“Andrea, I understand your concern,” Bill offered. “Michelle makes a good point. Jason and Michelle, for our next huddle with Susan, let’s bring this up. We may need a small team in the short term to help out with this. Let me be clear, I don’t want to throw the baby out with the bathwater. What we saw here today was great. But I’m under no impression this is the end all be all, although we are delivering what we set out to deliver. I’ll volunteer to help you with the internal issues. After all, this is a product, right? So it will need a product team.”

Clapping started to fill the room and everyone started looking around. It was Jason.

“Well done, Team Kraken!” Jason said. “Bill is right. There is more to do. But I don’t want us to miss this golden moment. We just leveled up! And, Omar, great demo.”

Omar blushed a bit. The rest of the team smiled and joined in on the applause. It had been a difficult past few weeks, and it was really nice to hear some appreciation from IUI leadership.

It was clear from the faces in the room that the team felt a new surge of energy. They could do this!

In our next installment, just as Omar’s policy engine insights seem set to unite Team Kraken’s security and engineering pioneers, impatience reigns! The suits fume over Omar’s perceived indifference while Michelle’s voices visions of smoother software and compliance building, if only wasteful red tape would give way! But with less runway left, will leaders grasp Michelle’s dreams? Join us next time for the continuation of the story. Or, go to your favorite book retailer and pick up a copy of Investments Unlimited today.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•