Turbo Eureka Is Born – Investments Unlimited Series: Chapter 6

Welcome to the sixth installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

In our last installment, the first rays of collaboration were beginning to shine through. In this chapter, Omar’s demo dazzles but leaves auditor Andrea uneasy—what’s missing? The crew must sell their vision past wary execs, so Michelle attempts an ingenious “Three Lines” model to bring security seamlessly in-house! But will this unprecedented plan fly?

Tuesday, April 26th

Over the next several days, Bill and Michelle worked on a presentation outlining their proposal to move to an automated governance structure. They needed to get sign-off from Tim, Jada, and Jason before taking their proposal further up the chain to Susan and the board, and then submitting to the regulators. Michelle had already walked through some of the policy ideas at the initial findings meeting with Tim and Jada. They had been hesitant, so she hoped having Jason in the meeting with them would help smooth things over.

Luckily, since addressing the MRIA was top priority at IUI, they were able to get a meeting on the books quickly. Jason was excited by the idea and eager to see how it would be implemented, but others were hesitant.

“I don’t know, Jason. Let’s see what Laura and the outside audit team think before we sign off,” Jada said.

“Yes, I want to think about this some more myself,” Tim replied.

“We need to move on this quickly, so give me your final thoughts within the next two weeks. Then we’ll fill Susan and the board in on our proposed direction. If they approve, we’ll still need to formalize our proposed approach to the regulators and see if they approve it before our ninety-day deadline.” Tim and Jada nodded, then Jason turned to Michelle.

“Once we have approval, you’ll need to get started on an engineering solution. Pull together a team and let me know what you need to help pave the path forward.”

Michelle nodded. “I have just the team in mind.”

“Great work, everyone. I’m excited to see how that plays out, but remember, we’re not even close to being finished here. I’m sure we can expect a long review process with the regulators while we deliver on our proposed promise. They’re not going to just take our word for it, after all.” Jason laughed. “Get ready for some good old-fashioned waterfall delivery dates and RAG status updates!”

“Oh, goody,” Michelle replied with false enthusiasm. 

RAG updates, those project templates with red, amber, or green stoplight icons indicating the state of the project, were a joke. They took hours of time to assemble but were full of subjective guesses and spin. 

“I’m always ready for some good old watermelon reports,” Michelle replied sarcastically.

“Watermelon reports?” asked Tim.

“Yeah, you know, waterfall delivery status updates that show green on the outside but are red on the inside,” Michelle explained, causing even Jada to crack a smile. “I’ll bring some crayons.”

“That’s the attitude!” Jason smiled and held his hand up in the air for a high five.

Michelle looked around and then, seeing that no one else was reaching forward, raised her hand toward Jason’s. He slapped her hand with genuine enthusiasm before ushering them out the door. Not for the first time, Michelle was left wondering how Jason maintained his seemingly never-ending positivity.

Thursday, April 28th

Susan was staring out the window. The early morning sun was bright and painted the Charles River with a shimmering pool of light. She could hear Bill, Carol, Tim, Jada, and Jennifer talking behind her during their usual Thursday huddle. They were discussing the MRIA details and the automated governance response to the regulators that Bill and Michelle had put together.

“Excellent,” Susan said as she turned around to face the team. “Are we ready to give this to the regulators?”

Jason jumped in. “It gets my vote! Not only should this satisfy the auditor, this automated governance proposal will better position us to maintain velocity on our digital efforts while respecting the guardrails.”

“But I think there is still more to do,” Jada said, adding caution. “Can the engineering team deliver or do we need to get them some help?” 

Tim nodded in agreement.

Bill groaned. He had expected to hear another campaign to hire an external engineering consultant to fix all their problems. “I know it is going to be hard work, but I trust the team. We got this.” 

Jennifer nodded her head, as did Jason.

“I agree. I trust our team,” Susan said, looking at the team then back out the window. “We’ll update the board and the regulators on this automated governance plan. Bill, continue working with your team on the engineering solution and keep me informed on the progress.”

Friday, April 29th

While the senior execs were working on passing their proposal through the approval process with Susan and the external audit firm, Michelle and her team, nicknamed “the Kraken,” took over a section of “the dungeon” so they could all work together as they built out the automated governance solution.

There wasn’t actually anything dungeony about the location. It was well lit and bright, but there were no windows to the outside world, hence the nickname. The teams had embraced the windowless cavern and themed the place as a medieval fantasy land. Cube walls were dressed to look like stone. The engineers were encouraged to make this space their own. Some decorated the walls and tables with dragons, swords, and jewels. It definitely added fun to the otherwise dull environment, and the team could use that about now.

Michelle had decided to hold a kick-off happy hour that afternoon as a way to build team unity. Even though this particular group had worked together before, Michelle thought it would be good to get everyone aligned to this special project. They were going to need to stay focused over the coming months if they were going to help get IUI through this storm.

“Great party, boss,” Dillon, one of the site reliability engineers said, walking up to Michelle. “But maybe we shouldn’t call it a happy hour anymore. I only see two people drinking anything alcoholic.”

“You’re right. But I think the name has stuck. Besides, you can attend happy hour without booze!”

“Hey, speaking of names, what are we going to name this project anyways?” Omar asked, and the whole group turned toward him, nodding their heads.

“I’m not sure any Greek names are available. It seems the Kubernetes community has commandeered every Greek word for their projects or products,” Michelle said, laughing slightly. “What else could we do?”

“Whatever name we choose, make sure we can spell it easily,” one of the other engineers said. “One of my friends launched an open-source project a few years ago from their company and no one could spell it right. It was a disaster.”

With that, it seemed like every engineer pulled out their phone or tablet and started searching the web for Egyptian deities, Celtic gods and goddesses, and more. Nothing seemed quite right.

“I have an idea!” Omar interrupted. “Our source code repo generates random names. Let’s pick the first name it generates.” Everyone looked at each other. No one was in disagreement. Omar set aside his drink, pulled up a browser on his computer, and navigated to the source code repo. After a couple clicks and pounds, he said, in a horrible English accent, “I hereby pronounce thee ‘Turbo Eureka.’”

“Turbo Eureka? That sounds like a vacuum cleaner,” Dillon noted.

“I think it has an aura of cheeky prestige,” one of the other staff engineers, responded.

“Turbo Eureka it is!” Michelle announced, lifting her glass of IPA into the air.

“Hear, hear,” everyone cheered.

Michelle smiled. This was a great way to start. She just hoped they all kept up this level of enthusiasm for the project over the coming months. She had a feeling that this was going to be the most challenging project she’d ever worked on.

Michelle had spent almost all of her professional life in the finance industry, and she was all too familiar with how organizations dealt with regulations. She had experienced the same situation time and time again, where detailed system requirements became policy, and the policies kept growing and growing. At one point there were too many policies to keep track of, and people started ignoring them. No one cared. No one even knew where the actual policies were stored. Now it was just called bureaucracy and red tape.

As Michelle pulled her car out of the IUI parking lot later that evening, she reflected on how execution was tightly tied to the interpretation of these policies. It seemed as if the implementation begged the architecture, when in reality, the architecture of the system should drive the implementation specifics.

She seemed to experience post-traumatic stress when “the business” signed off on requirements. Without fail, there would be a need, not too long after the sign-off, to make a change. Some aspect of the system operations, development, process, or security would need to change. Michelle and her team had measured how much time it took to complete all the required paperwork for the change approval board. They were astonished when they calculated it to be just under five hundred person-hours.

Five hundred hours of pure work effort; not accounting for wait times when an email or feedback was holding up completing a change board requirement. The engineers constantly lamented how subjective the process was, joking that if they were to give the same software and policy to two different change approval board members, they’d get completely different answers.

Michelle merged onto the highway. As she drove down the congested road, the lanes of cars reminded her of IUI. Each lane reminded her of an organizational silo. These silos at IUI were being broken down by forced collaboration, visualizing work, and automation. Even with these advances, software delivery did not noticeably quicken.

She chuckled out loud as the person in front of her signaled they were going to merge into the lane to their right. Even though the car put on its signal and seemed to meet all the proper cooperative driving expectations, the merge itself slowed down traffic. Michelle started to think. Even though we come up with processes and an enhanced understanding to speed things up, we never go any faster—we just create slowness elsewhere.

Michelle realized that there was a fundamental flaw with the rigor of the change process. This rigor was not based on how the system needed to operate, it was based on what happened historically. The more severe the MRA, the more arduous and long-lasting the repercussions. It was as if no one cared what the underlying cause was; they just wanted to address the symptoms.

Addressing the symptoms as they exposed themselves was the catalyst for an ever-slowing software delivery process. It was always in the name of security and risk. More and more processes were created, more complexity was added to the systems, and more time-wasting meetings were required. It was like organizational scar tissue. It kept building up, slowing down the processes even more and frustrating everyone involved.

Many people attempted to standardize much of this overhead. They were never successful because of the differing ways the engineers operated and their inability to agree on a converged set of operation approaches. All of this resulted in an exponential increase in time and resources to build and operate their applications.

“Holy moly,” Michelle shouted as she slammed on her brakes. A car ahead of her merged into her lane with no turn signal. As she started to pay more attention to the road, she saw it happen again and again. For a moment, it seemed as if the norm was for other cars to force their way into the spot they wanted to be in as opposed to using their turn signals.

This aggressive driving reminded her of policy exceptions. There were a lot of people who forced their way through the change board using policy exceptions. After all, policy dictated what was required for any change to occur. Either by sheer force of will or by someone with a larger political clout taking responsibility, teams veered in and out of production with policy exceptions.

Escalate everything! She remembered a lunch meeting where an IUI business leader bragged about how they were able to get so much done at IUI. “If someone or something is blocking you, don’t stand for it: drive around, fight for it, escalate.” She laughed to herself thinking about how Barry and Andrea would react to that.

As she turned off the freeway, she started thinking about the DevOps Automated Governance Reference Architecture paper. There was something about it that was brilliant, but she just couldn’t put a finger on it.

As she pulled up to a stoplight, it hit her. “The DevOps automated governance takes subjectivity and makes it objective,” she said as if she was talking to an invisible person in her passenger seat.

Michelle realized it wasn’t necessarily the concept of a change board that was the issue; rather, it was the subjectivity of how the change process was managed that was the major contributing factor. Subjectivity was the reason two differing change board members could give a multitude of differing, and even conflicting, answers. The change process was not a process with a set of objective specifications. It was a process of people passing or failing software based on their own warm-and-fuzzies.

Maybe this was the reason for the lack of transparency, she thought to herself. You can only be transparent when you have something to point to. If you don’t have an objective measure, you don’t have anything to point to.

The organization depended upon the subjectivity of its people as opposed to objective measures. They were opinion-driven, not data-driven. “Am I the only one to think about this?” she said to that invisible passenger in the front seat.

Michelle had a breakthrough moment as she pulled into her driveway. We trust our compliance outcomes to people who have the authority to make a decision. At best it is based on some policy and at worst it is subjectively based on anything else that could influence that person, she contemplated as she unbuckled her seatbelt, grabbed her laptop bag and purse, and then opened her car door. That’s ridiculous, she thought to herself. No wonder it’s so unclear, inefficient, and painful.

Walking into the house, she continued to challenge her own thoughts. If we can automate DevOps governance, what unexpected consequences could there be? Even if we are doing everything correctly, making clear and great documentation and abiding by policy, is there somewhere we could be misinterpreted?

Michelle didn’t want to be doing all the right things and still be viewed as violating policy. She knew the smallest infraction or policy violation would create even larger obstacles to broader adoption. Change at these financial institutions was a long process. The length of change multiplied, and further encumbrance by a mountain-out-of-a-molehill perspective of the slightest misinterpretation could render these new efforts moot.

In all these thoughts, Michelle realized she had only been thinking about the change board, Security, Compliance, and auditors. She hadn’t fully considered the engineers themselves. The fastest way to upset an Engineering team is to undertake a more intense process to develop and deliver their software. She realized how essential it was to ensure that any automated change process was decoupled from the engineers. 

This automated process must ensure that the existing feedback loops established are not altered in any negative way, she thought. As physicians say, “First, do no harm,” she joked to herself. And if this automated governance results in reduced friction for engineers, all the better!

Michelle set her bags on the dining room table and walked over to the kitchen. She could hear her twins and wife laughing in the backyard. She walked to the back door and looked out on the twins running around. As she watched, she realized the kind of rock-and-hard-spot situation the team could end up in if they weren’t careful to design the approach that considered all stakeholders in the IUI software development and delivery process.

Monday, May 2nd

Standing at the high-top tables in the Kraken’s section of the dungeon, Michelle kicked off the first official meeting of project Turbo Eureka.

“Alright everyone,” Michelle said, raising her voice high enough to ensure the whole team could hear. “We just received word that the execs presented our response to the MRIA to the regulators. But that was the easy part, if you can believe it. Now we need to fulfill our promise, and we have nine months to do it. We’ll be having monthly demos and check-ins with the regulators, and we need to keep them up to date on our RAG status.”

Michelle heard a few groans around the room. The Kraken team had been used to working in an Agile/DevOps manner for some time, being one of the first teams to migrate during the early days of IUI’s transformation. Working in a waterfall manner with RAG status updates felt like a step backward.

“If we were to build a thing,” Michelle said, looking around the room at her team, “a thing that would automate the whole process so that an audit like this MRIA would never happen again  .  .  .  how would we do it?”

Omar leaned in. “We’d start with the first control: peer reviews.”

“Is that the easiest?” Michelle wondered.

“I don’t know yet. But at least I know where to start. We already use Git to automate a lot of our daily workflows. Adding another set of API calls won’t take us long at all,” responded Omar with an easy shrug. “I’ll mock something up. Just give me a day.”

“Okay, let’s start there.”

A few days later the team joined up again.

“Omar, let’s see this prototype that you’ve been talking about all morning,” Michelle said.

Omar stepped up to the board. “Okay, so check it out,” he said as he gestured to his screen. “Um, can someone approve this pull request?” He forcefully clicked his mouse.

“Done,” said one of the other engineers, giving a thumbs-up in Omar’s direction.

Omar moved his mouse toward the Merge button, which would join his changes with the main code base.

“Watch my console,” he instructed as his eyes darted to the bottom left of the screen. In twelve-point mono spaced font, a green word was illuminated against the black background: PASS.

“See that there? There’s the attestation. I even added color to the logging!” exclaimed Omar. “And see, it’s in the database too.” He tapped his keyboard once, revealing evidence of his claim in the console of his local database client.

“That’s slick,” Dillon said, nodding his head in approval. The rest of Team Kraken appeared equally impressed.

“Great work, Omar!” Michelle cheered. “That was quick. Walk the rest of us through how you achieved this.” Michelle sat back and admired the way the team listened attentively to Omar’s explanations.

The engineers in Team Kraken were known for their fast turnarounds. They typically worked in two-week sprints that ended every other Thursday with internal demos. These demos provided the team with an opportunity to show off their new features and receive feedback. Michelle loved these meetings because she and her team could see the reactions of their stakeholders when they finally had the chance to show off all of their hard work.

Michelle was invigorated by Omar’s prototype. It wasn’t much, but it provided a spark to a seasoned engineer who had grown accustomed to a mundane cadence of bug squashing and incremental feature enhancements. In Michelle’s mind, this was an opportunity to engineer an entirely new set of capabilities that, if done right, would empower IUI engineers to operate at full speed with the confidence of knowing that they’re in full compliance.

Next up, Jada lauds potential but balks on ceding all control – implementation remains foggy as a livid Jennifer threatens outside help! Can the coalition hold together with the finish line in sight, but concrete action is still hazy? Signs point to a pivotal Team Kraken showdown ahead! Join us next time for the continuation of the story. Or, go to your favorite book retailer and pick up a copy of Investments Unlimited today.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Director, Global SRE @ Disney | Speaker | Co-Author of Investments Unlimited

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•