The Final Countdown – Investments Unlimited Series: Chapter 13

Welcome to the final installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

This is it…the final countdown! After a grueling year of inter-departmental clashes and executive clashes, while constantly calibrating complex software intricacies against compliance concerns, Michelle’s vision still remains controversial! Can a final, climactic boardroom showdown secure urgently needed autonomy and resources? Or will Tim and Jada’s old guard muscle in, fatally undermining fragile progress so far? Long nights now loom for our legion of irregular technologists. Stay tuned for the final nail-biting chapter of Investments Unlimited!

Tuesday, February 7th

Carol and Michelle walked into the conference room. All the bosses, and the bosses of the bosses, were there, including one of the representatives from the external audit firm, Laura Perez. Even Jason was in attendance. Tim and Jada were sitting in the front two seats of the table, right across from each other. Tim’s back was facing the windows. It was a nice day out. Bright, sunny, and not a cloud in the sky. But inside Michelle could have sworn she saw a storm gathering overhead. This meeting was going to be anything but smooth sailing.

Michelle set her things down and tried to look calm. She scanned the table, noticing Jennifer next to Tim. Jason had spun his chair around as soon as he heard them walk in.

“Well, the stars of our show!” Jason said excitedly, briefly clapping his hands.

As his clapping settled down, Jada stood up.

“Not that we need a reminder, but let’s start with why we are here today. Nearly one year ago, we got served with an MRIA from regulators for, and I quote,” Jada read off the paper in front of her, “‘Inconsistent process, ineffective in ensuring security and compliance, resulting in unauthorized and vulnerable software with significant amount of defects being released to production.’

“Over the last ten months and change, since we were first hit with the MRIA, we have made a lot of advancements. And we’ve made a lot of mistakes. But today, IUI is barreling down a new path to delivering software in a way that includes everyone who has a concern with the software we write internally.”

Jason clapped again. Michelle looked at him—clearly he wasn’t feeling the same stress as she was. What did he know that she didn’t?

“What we will walk you through next is the core of our strategy to resolve these findings and prevent—er, significantly reduce the likelihood of similar future findings. With that, I’m going to turn it over to Bill, who’s been working with a team of engineers, Security, and Audit to craft our response and the future of governance here at IUI.”

Bill stood up. Michelle swallowed. She, Omar, and the whole team had been working for so long on this project that she felt like she was watching her own child stand up to present their PhD dissertation. She begged the butterflies in her stomach to settle down.

“We’re not here today because we failed an audit or were hit with an MRIA; we are here today because we failed to think bigger and better about how we build software,” Bill said. “In retrospect, we did a lot of things right as we transitioned to a DevOps way of working, but we didn’t do all the right things. That’s why our software engineering process failed us: we only considered Development and Operations, Dev and Ops, not Security, Compliance, or Risk. This was our big failing.”

Once again, Jason clapped briefly, nodding his head in agreement.

“Our problem was that we did not have proper controls in our software development process, hence all the MRAs leading up to the MRIA. How did we not catch this? These findings could have been preempted if software engineering had engaged our Risk teams in order to build out our software delivery workflows to include the proper controls. Why did we not do this? Well, simply put, it was not how we thought about DevOps. We thought speed of delivery trumped everything else. We failed to include security and risk as features of our product. This lack of early inclusion and continued cooperation leads us to the primary issue.

“I am now convinced that this primary issue was a misguided perspective for assessing what the requirements of our software development and delivery process should have been,” Bill explained.

Jada commented, “I think we all have a bit of a misguided perspective there too. We’ve talked about more cooperation with the Engineering teams, although we’ve been at a loss on how to make that happen. Being explicit partners and contributors to the design of the software development process is a great approach.”

Another series of small golf claps could be heard from Jason.

“Yes, well said,” Jason chimed in. “This is an example of the three ways of DevOps: flow, feedback, and continuous improvement. This type of collaboration between teams establishes flow, and flow is about ensuring that defects are not passed to the end user. We can look at risk and compliance with either the software delivery process or the software itself as a quality validation. This is a profound understanding of how the IUI ‘system,’ and I say that broadly speaking, could work.”

Everyone at the table now had a light smile.

Michelle smiled. Wow. Are we actually learning and growing together? Maybe this won’t be such a bad meeting after all  .  .  .

“Okay, on to our guiding policies,” Bill started. “Think of these as the guardrails. A guiding policy does not tell you the actions you must take. Instead, they give you the right amount of information to choose the actions you should consider to solve the problem. They channel the action into a specific direction without explicitly stating what should be done. First guiding policy: if the rest of these policies are abided by, then you can bypass the IUI manual change approval process and go straight to production.”

With those words, you could practically hear the proverbial record scratch. Jada and Tim’s faces went from excited to concerned. This proposal had been in front of them in the past, but neither of them wanted to touch the topic. They had heard it from the developers earlier; now they were hearing the same from product management. But more than one IUI team members’ careers had focused on the manual change approval process. The CAB was at the center of every change that went into production. Everyone dreaded going in front of the CAB and presenting a change request form. The notion of skipping the IUI change approval process was very disruptive.

“Jada, Tim, your faces say it all. But think of it like TSA Pre-Check: less risky passengers get to move through the process faster. Let me finish real quick, and then we can talk about your concerns. I bet we’ll address them with these other three policies.”

Both Jada and Tim looked at each other, then at Bill, signaling permission to move forward.

Bill glanced at the slides and continued. “Second guiding policy: complete automation must be implemented for capturing evidence of quality, risk mitigation, and compliance for software and its delivery process. The only manual process is peer review, which is a must when software is being designed and developed.

“Third guiding policy: security and compliance requirements are as important as functional requirements, and hence all software product teams must involve Security, Risk, Compliance, and Audit teams to identify these requirements from day one.

“Fourth guiding policy: the software budget. A budgeting system, similar to a financial budget, will be established to track our deficit in quality, risk, compliance, and audit. When a budget has been exhausted, the team cannot work any new features and must pay down the debt completely. This budget shall be available for anyone in the company to see at any time,” Bill said before taking a deep breath. It seemed like the first one he’d taken since he’d begun.

“Wait, why won’t every team be 100% compliant all the time? Why would we ever want to allow anything less than 100%?” Tim continued, “Gee  .  .  .  don’t we already know why we are here?”

Carol replied, “While we want to always achieve 100% compliance with any rule we set out for the organization, the reality is that that’s largely unachievable. When we set our rules, we have no feedback process for how arduous it is to achieve them; the organization just assumes they will be met because ‘it’s the edict.’

“In practice it’s quite the opposite. Most will try to meet the expectation, although when a conflict comes up that pulls them in another direction, many will rationalize their work as ‘good enough.’ We can’t really empirically track what is and isn’t in compliance with rules.

“Taking this new approach acknowledges risk and provides a mitigating framework for addressing it empirically. The product teams will be empowered to make the trade-off decisions since they have the most context. Leadership, such as yourself, can set the specific budget limits. This now makes quality, risk, compliance, security, and audit the responsibility of the product team.”

“So you are saying that we could release a new version of a software with a known critical vulnerability? And we will allow product teams to make that decision?” Tim asked.

Jason quickly interrupted with a small whoop. “Security is the responsibility of the engineering team! Decision-making is closest to those with the most context! Yes! Congratulations, Carol; you sound a lot like Shannon Lietz. I think she was one of those who coined the term ‘DevSecOps.’ Her foundational idea was to make security the responsibility of those who are actually building the software, not some other folks who have little knowledge of the software and business problem. It’s the epitome of bringing the authority to information, not the other way around.”

Michelle had been trying to interject for a while, and finally she was able to get a word in edgewise. “I can give you some examples. Let’s look at two examples of compliance policies: policy number one, ‘no software releases are allowed with a known critical vulnerability,’ and policy number two, ‘unit test code coverage for new code must be at least 60% for a new release.’ I can confidently say that no one would release critically vulnerable code to production, but the team may decide to move ahead with, say, 55% code coverage and commit to bring it up during the future releases. No matter what the teams do, we will have the evidence of what they did, and I think that is what we were lacking before all this.”

“Okay, this is compelling. I need to get a bit more comfortable with the idea, but I see where you all are coming from,” Tim said.

Jason turned to Tim. “What the team is proposing has similar fundamental concepts to those that underpin SRE. Where SRE is focused on enhancing the software, or software development process, for reliability and for end users, this fourth policy is enhancing the software and development process for compliance and security. Heck, if we do this right, we may be able to write a Compliance and Security Engineering book like Google did when they published the SRE book!” Jason looked positively giddy. Not for the first time, Michelle wondered where his source of unending positivity came from.

“The CSE book!” Tim smirked.

Another click was heard and the second-to-last slide of the presentation appeared: “Next Steps.”

Bill looked at Michelle, Carol, Barry, and Andrea.

“We’ll take it from here,” Michelle said. “The following is the dashboard we will be working with.”

On the screen were the words, “Why would we not want this?” The slide featured a dashboard that broke down pull request approval, unit test code coverage, static code quality, and more.

“But wait, there’s more,” Michelle continued. “Look at how we now view our system down to the component level.” She advanced to the next slide.

“I want this,” Jennifer said.

Not even a tenth of a second passed before the external auditor, Laura Perez, said, “Wow! Yes, me too. I don’t think I’ve ever seen something quite like this before. In fact, I’d be very eager to talk with your team more about how we might use this example to help other organizations achieve this. Maybe even present a joint experience report at a conference or something.”

Michelle beamed. She couldn’t wait to tell Omar and the rest of Team Kraken how well their work was being received.

All eyes turned to Jada. She was still in deep thought. Her five-second pause felt like five minutes. It then became clear that Jada was awestruck.

“Yes, I want this! I want this, like, yesterday. Let’s face it: we’ve told ourselves that we’ve had a plan for reducing risk in DevOps forever. But at best  .  .  .  at best we’ve had three different plans: one in our Engineering teams, another plan from our Cyber Risk folks, and another from Internal Audit. Now, for the first time, automated governance will ensure all three functions will all be aligned to the same vision of what it means to reduce risk as we build software for the business.”

Tim nodded. “Yes, this looks better than the last demo I saw. I think it’s clear to say that what your team has built is a game changer, not just for IUI but for the community at large. But let’s talk about onboarding. How many teams are now using this at IUI?”

Michelle’s stomach flip-flopped. This was the moment she was dreading. She pulled up another screen, which showed a step-down ladder of the TLCs at IUI. Michelle walked everyone through the onboarding process to date, including how they had used the “checkout” stage in the pipeline to automate onboarding before they discovered the plateau in adoptions and the AppRails platform.

Michelle steeled herself. “With the discovery of this legacy platform, we’ve essentially hit a roadblock.” Michelle avoided Tim’s eyes and went on. “We’re proposing dedicating the necessary time and budget over the next quarter, at least, to helping them onboard. In addition, there are other legacy apps that will be too costly to try and migrate over. Instead, we are proposing maintaining a manual process for these  .  .  . ”

Michelle trailed off as murmurs and comments began flying around the room.

“What good is this new automated system if half the organization doesn’t follow it?” Tim said.

“This isn’t going to go over well with the regulators. Once again, we’re not meeting our burden of security,” Barry said.

“I agree. Susan and the board aren’t going to like this,” Jennifer said.

At that moment, just as it looked like the meeting was going to dissolve into another blame game, Jada stood up.

“Everyone calm down. Now, Michelle and her team have achieved an amazing amount in the last year. Let’s not overlook the enthusiasm you were all just showing minutes ago when she displayed the new dashboards.”

“Yes, the dashboards are great. But it doesn’t do us any good if they’re not being used,” Tim said, obviously exasperated. He turned to Jada. “Do we think we can move forward with anything less than 100% onboarding? I’m not sure.”

Michelle felt her cheeks turn red. She simultaneously felt ashamed and frustrated, a feat she didn’t know was possible.

“One hundred percent onboarding is just not feasible,” Michelle responded. “And it’s even irresponsible in some cases. Some of the legacy apps would spend more time and money trying to force their square peg into our round hole than if they just keep doing what they’re doing. Better to let them live out their days comfortably achieving manual compliance until it’s time to sunset them.”

“But that’s putting IUI at risk of even more audits and another MRIA!”

“Actually, I don’t think it is,” Jada responded. “Michelle and her team brought their concerns around onboarding to Audit so we could work this problem together.” Jada turned to Michelle and Andrea and smiled. “And rightly so. As we dug into the problem, we went back to the original findings of the MRIA and our own internal audits. 

“Now, if you’ll remember, the MRIA states that IUI was found to have ‘inconsistent process, ineffective in ensuring security and compliance, resulting in unauthorized and vulnerable software with significant amount of defects being released to production.’ Nowhere in this audit do regulators state how we are to correct this. There’s no requirement to onboard every app at IUI onto an automated governance system. As long as everyone at IUI is doing what they say they’re doing, and we have the evidence of that, we’re in compliance.”

“Automated governance was our choice and probably the only way out to continue on our DevOps journey,” Michelle added.

“What about the original TLCs called out in the previous MRAs?” Laura interjected. “Are they on this new automated system?”

“The majority are, yes,” Michelle answered. “All except the TLCs on the AppRails platform. But those are following the manual policies we’ve put in place and spoke about at the beginning of the meeting.”

There was silence in the room. Everyone felt a little uneasy having this discussion in the presence of an external auditor. Michelle wasn’t quite sure if she felt like standing up and cheering at how far they’d come, or slumping in her chair as she thought about a year of work and what some might think is very little to show.

All eyes turned to Laura from the external auditor firm. Ultimately, the board was most likely to agree and follow whatever she and her firm said.

“Mmm hmm,” Laura responded, nodding her head. “I think Jada is correct. The MRIA doesn’t require how IUI addresses the concerns, just that the concerns are effectively addressed. Whether some sections of the organization are in compliance manually or through the automated system shouldn’t make a difference. I think that with what you’ve put in place here, and with a clear plan for migrating or addressing these controls in the future, you should be able to convince the regulators and close out this project. Remember, the regulators’ biggest concern is that you show you are actively reducing risk.”

Michelle held her breath, looking around the room. Tim had leaned back in his chair and was staring at Laura. Jennifer had her head cocked in an appraising way. There was silence for what felt like an eternity.

Then Michelle was startled to once again hear clapping. She turned to the back of the room. Jason was now standing and clapping loudly. The rest of the room had turned to face him, many with quizzical looks on their faces.

“Congratulations! You’ve done it,” Jason announced.

“Done it?” Tim said.

“Yes! I’m convinced. And so is Laura, I take it.” He turned to her and everyone could see her nodding in agreement. Jason continued in a calm voice, “Look, nearly a year ago, IUI almost came to a full stop, and its future existence came into question. Susan had only two choices: to give up or to trust her team. She picked the second. It was unfortunate that it had to come down to that. It didn’t need to be that way!”

Everyone was looking at Jason’s face. Tim wasn’t sure what to expect.

“Just like all other companies, we embarked on a DevOps journey over the last few years. Just like all other companies, we started embracing the goodness of DevOps across our teams. The leadership was proud of what was going on. And then the MRIA shook our world.

“We had all the right people in the right places. They just needed to come together to create the magic. That should have been simple, right? But it wasn’t. Teams didn’t come together. We eventually ended up in a place where we even questioned our own existence or ability to survive.

“Over the last year, everyone in this organization felt the pain, yet they went above and beyond. I applaud our engineering teams that looked outside the box. They didn’t shy away from experimenting, testing out innovative solutions, and even failing multiple times. But they did not give up!” Jason looked at Jennifer, Carol, and Michelle and clapped once more. “We even had one of the worst supply chain vulnerabilities hit us during this process, and this team took care of that too!

“To our Risk and Audit partners  .  .  .  thank you! Thank you!” Jason was genuinely appreciative of how Jada and her team stood up to the cause. “Tim and his team also supported us during all this. So, thank you, Tim, and everyone from your team as well.

“Finally, I’ve been thinking about this for a few days now. I’m asking myself if we are way better off now when it comes to software development security, audit, and risk management.” Jason paused for a moment, looked around, and continued. “I am now convinced that if we aren’t using DevOps principles to embrace continuous integration and continuous delivery practices, we aren’t going to be compliant enough. ”

Jason paused, smiled broadly, and looked around the room once again. “I am so proud to be part of this team. You should all be proud too! I believe we just marked an indelible turning point for IUI. And you and your teams made this happen! What do you think, Jada? ”

“I agree! Well put, Jason,” Jada replied. “Very well put. Great job, team!”

Jason began clapping again and everyone joined in. Michelle smiled and felt goosebumps. She looked at Carol, Bill, Tim, Jada, and others in the room. They were all smiling. Even Barry had managed to muster a grin as he joined in on the applause. Incredible! Michelle thought to herself. We actually did it.

As the clapping began to subside, Jada concluded, “Now, unless there are any other concerns, I think we can adjourn.” Jada paused to look around the room.

Laura gave a shake of her head. “I have no objections.”

“Excellent,” Tim said. “I think we’re ready to present our new processes and procedures to the board. Carol, Michelle, you and your team should continue your work onboarding TLCs. We may have pushed obstacles off our path and paved a road for the future, but that road needs constant maintenance. Our next step is going to be ensuring IUI is never put in this situation again.”

“Hear, hear,” Jason said. “Well, if you’ll excuse me, I’ll just go and fill Susan in on the great work you all have done.”

Jason left the room and the other execs began gathering their things. Michelle picked her things up as well. She couldn’t quite believe they had made it through.

—

“Thanks, Jason,” Susan said grinning from ear to ear. 

Jason had just updated her on the progress and was joined by several from the leadership team. 

“I’m incredibly proud of what the team was able to accomplish,” Susan continued. “And, this Michelle, she seems like someone we would want to invest in.”

“Absolutely!” Jason exclaimed, adding, “Jennifer’s team did a great job but Michelle was the Sherpa that got us over the mountain.”

“I agree completely, she’s definitively a leader we need to promote,” Jennifer added.

“She even impressed me,” Jada said, “and that’s not easy to do.” 

Jason laughed and stood to leave, “We should make sure Michelle gets the recognition she deservers. Heck, the whole team should be recognized! I suggest Jada and Tim kick it off with a karaoke party. I hear you two are hidden stars!”

Jada’s mouth dropped open. “How did you know?” 

“Don’t worry, I won’t bring witnesses,” Jason said.

Everyone laughed.

Epilogue: Tuesday, March 4th

Susan was sitting at her desk, reading through the latest book Jason had given her to read, when her phone buzzed. She picked it up and saw that it was Bernard.

“Susan,” Bernard’s voice rang from the other end as she answered. “Just heard back from the examiner. Looks like your team pulled it off.”

“That’s excellent news, Bernard!”

“This is great work. The whole board is feeling confident.”

“Thank you. I’ll pass that news on to the whole team. Have a great weekend.”

“You as well.”

Bernard hung up, and Susan placed her phone back on the desk. She packed up her laptop, the book from Jason, and various print-outs of papers and articles Jason had sent her, all on the future of technology businesses.

She walked out the front doors of IUI. I just love Boston in the spring, she thought to herself as she made her way past the towering buildings. She glanced over to drink in the view of the Charles River Esplanade. The air was crisp and cool but soon the trees would be budding and painting the glassy river with a brilliant green glow. She smiled. It reminded her of all the reasons she had relocated to this area in the first place.

There was construction in front of the neighboring building. Large water pipes, power conduits, and steel reinforcing bars were all exposed. Susan thought to herself how soon it would all disappear, and in its place, a pleasing landscape and facade would emerge. It struck her, reflecting back on all that had happened at IUI, that technology and security are like those buried elements. When done well, they can’t be seen from the outside. They’re invisible to the customer and even to the company itself. But if they’re not included in the business strategy and execution, then the business would be missing critical flow, power, and structure to make it all work. Things may appear to be going well, but just under the surface there would be problems.

She took a deep breath, feeling the weight of her briefcase filled with books, articles, and papers. If anyone had told her two decades ago that as a CEO she would need to become an expert in technology and security, she would have laughed. Sure, technology and security are important but they’re like utilities, cost centers in service to the business. Leave it to the smart nerds and paranoids in the basement. Right?

She realized now how misguided that thinking was. As Jason had reminded her, not only is technology the lifeblood of the business, it is key to amplifying and accelerating everything they do. By keeping it at arm’s length, they had inadvertently allowed it to lose connection with the business. Security, audit, and process gaps formed and had been amplified. However beautiful their shiny business might be on the outside, cracks had formed, and they suddenly found themselves hurtling toward disaster.

A cool blast of air hit Susan as she rounded the corner. She looked up and saw dark clouds starting to gather in the sky. City noise filled the air and washed up against the office buildings as business travelers made their way home from work. There is always a buzz and energy in the air here, Susan thought to herself. She wondered how many of her fellow travelers were likely to face the same challenges she and IUI had just gone through, and how many more challenges waited in IUI’s future.

The street was lined with office buildings. Each one seemed to try to stretch just slightly taller than its neighbor. Capturing business in this competitive landscape demanded innovation, speed, relevance, and security. Each layer of technology was like a floor in the buildings that Susan saw in front of her. Every layer prepared the foundation for the next, and the next, all the way into the future. For IUI or any business like it to survive and thrive, no separation can be allowed between the business and the technology that powers and builds it. She was convinced that now, more than ever, every business was truly a technology business and every business leader was a technology leader.

After speaking with Jason earlier in the day, her head was full of new ideas for IUI to try. All of them powered by new digital technologies. She felt a surge of excitement as she thought of all the possibilities. Anxiety tapped her shoulder, but she brushed it off. After all, she had the best team in the world. Sure, they would make mistakes, but as they had just experienced, she was convinced that they would be able to tackle any challenge if they worked together.

The sky’s the limit! She thought to herself, looking at the towering buildings in front of her. No, on second thought, there is no limit. We just need to have the courage to pursue it. Our future is truly unlimited!

Susan laughed. She was ready to change the world. But not tonight. Tonight, well, tonight was pizza night. She couldn’t wait to get home to Rich and Lucas and start building some of those pizza pies.

They did it! Jason lauds monumental efforts while smoldering execs soften, granting Michelle relief, momentum, and resources to remedy remaining rogue applications in time! What lies in store next as IUI eyes smoother seas empowered by peerless automated governance? Can they sustain this fragile coalition between code cutters and red tape regulators? Or will inertia, indifference, and infighting shatter this hard-fought new equilibrium? For now, though, raucous celebrations surely await as Michelle marks a checkered flag, capping her against-the-odds audit triumph! We salute you Team Kraken as you pass a pivotal milestone toward accountability and innovation! What gripping governance drama awaits in potential sequels? We’ll have to see! For now, though, kudos all around are long overdue! We hope you enjoyed this serialization of the book Investments Unlimited. You can pick up a copy of the book in ebook or paperback from your favorite book retailer.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•