Shifting Left with Risk – Investments Unlimited Series: Chapter 7

Welcome to the seventh installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

The last installment brought heavy executive heat as hand-wringing honchos warned outside aids may “rescue” the flailing Turbo Eureka system! But Michelle’s mavericks rally behind an ingenious badge prototype! Can they demo compliance evidence creatively before meddling management muscles in? 

Wednesday, May 18th

Over the next week, Team Kraken hacked away at Omar’s prototype. They had all been so inspired by Omar’s initial demo it seemed like everyone had ideas for features and capabilities.

During standup one morning, Omar asked the team, “What good is a record of pull request approvals if they can’t be linked to a deployable version of the application?”

There was a brief moment of silence.

“I see where you’re going,” said Michelle, “but how do we connect a merged pull request to an artifact that hasn’t been built yet?” She was referring to the continuous integration (CI) pipeline task of building and publishing an artifact to an internal registry. The CI pipeline ran after the pull request was merged, which meant that Omar’s prototype attestation had no awareness of a deployable artifact.

“Hang on, I have an idea,” muttered Omar. “Remember that pipeline plugin we built for last year’s all-hands meeting—the one that forwarded build metadata to a database for reporting?”

Michelle nodded.

Omar continued. “Those payloads contain the artifact metadata and the commit URL. We can use the commit URL to associate the pull request activity to the published artifact.”

“Yes!” Michelle said excitedly. “Let me add this to our backlog.”

Wednesday, June 1st

After a more than a week of hacking, interrupted by a long Memorial-holiday weekend, the Kraken team finally circled up to prepare for the demo, but Omar had one more surprise for them. After his initial prototype, Michelle had challenged Omar to find a lightweight alternative to his colored logging—a more visually appealing way to display the PASS or FAIL status messages without the overhead of a web UI. They had spun up this special task force quickly, pulling from an already established engineering team instead of creating a product team. This meant they had no UI dev onboard, and nobody on Team Kraken was up to date on the latest UI frameworks. Despite this, Michelle was confident her team would be able to pull something together that achieved their needs. Not for the first time, she thought about how IUI put together teams. She had recently read an interesting paper that was related to exactly this, arguing for full-stack teams instead of full-stack engineers, but that was for another time. She’d have to remember to forward that paper around the office.

Michelle joined the prep meeting and immediately noticed the huge large grin across Omar’s face. She figured he had an answer to her challenge.

“Omar, I’m guessing you have something for the team?”

“You’re gonna love this,” he said, commandeering control of the meeting screen. “You all know the drill by now; I need an approval on this pull request.” After receiving the required approval, Omar merged and kicked off the pipeline. After the pipeline finished, Omar clicked over to the demo application’s repository. “See the README?” he asked. Heads nodded around the room.

“Now, see that there?” Omar circled his mouse around the lower portion of his screen, which revealed three colored boxes: gray, blue and green. The green boxes provided the visual cue that the item had passed.

“That’s so cool!” exclaimed Michelle. “How did you do that?”

Omar, with a sense of satisfaction from the team’s approval, explained, “There’s this open-source project called shields.io that makes ‘badges.’ These badges can be placed anywhere that HTML renders, including README files. So I set them up to read our attestations and voilà! We’ve got a lightweight feedback loop for our demo.”

“This is one of the coolest ideas I’ve ever seen,” Michelle said. “It’s right there in your repository too—the developers never need to look anywhere else!”

Omar’s grin now stretched from ear to ear as he realized the added benefits of his clever idea. 

The first prototype automated the attestation of approvers on a pull request to the primary branch of a repository and correlated it with an artifact build and version. It was a simple process that was initiated by a webhook configured in the application’s source code management platform. A webhook is a common feature of most software that allows users to subscribe to events that occur on the platform. When one of these events occurs, it triggers a message to be sent to a given destination. Michelle had the system administrator of IUI’s source code management platform configure a global webhook to send all metadata to project Turbo Eureka’s receiver service whenever a pull request was merged.

The webhooks were reliable but didn’t contain enough information about a pull request’s activity. To help with that, the team engineered a “processor” that used the commit link in the webhook payload to execute an enrichment callback to the source code management platform. The callback gathered the history of activities on the merged pull request. After enriching the payload with pull request activities, the processor forwarded all the information to a service running a policy engine and used policy code to count the number of approvals from reviewers (other than the code authors) and provide a pass or fail ruling on the attestation. The pass/fail decision, along with the evidence, was then digitally signed (i.e., “notarized”) and stored in the database.

“This is great, team!” Michelle exclaimed, turning around. “I think we’re ready for our first demo with stakeholders.”

Thursday, June 2nd

“The purpose of this demo is to review the first actionable item,” Michelle started. The first demo to the larger team, including stakeholders from Audit and Security, was finally here. Bill, Barry, and Andrea were all in attendance and eager to see what she and the engineering team had been working on for the past couple of weeks. If all went well, they’d be using this demo in their next check-in with the external audit team and regulators. She used her table to project the MRIA Outline document onto the screen in front of them.

Actionable Items: Based upon the MRAs issued, the following items should be addressed with formal standardized approaches:

Goal: Define a minimally acceptable release approach

Objectives:

• Enforce peer reviews of code that is pushed to a production environment.
• Identify and enforce minimum quality gates.
• Remove all elevated access to all production environments for everyone.

“Enforce peer reviews of code that is pushed to a production environment,” Michelle recited. “Our team has been working for the last month and a half, and I think we’ve made great progress.” She turned to Omar. “Can you demo this feature in Turbo Eureka for us?”

“Yup, one sec. Let me take the screen from you,” Omar replied. A moment later, Omar’s screen appeared. All of his programs were dark, with differing bright colors for words. He had what seemed like hundreds of internet browser tabs open.

Andrea stared at the screen of code and laughed. “Is this the demo, all of this code?” Andrea inquired. “Because it all looks like Greek to me. I know it’s code, but I have no idea how to read it or even start making heads or tails of it.” 

“No, sorry, let me navigate to our source code repo.” After shuffling through his browser tabs as if it were an unorganized desk with papers all over it, the source code repo website came up. Omar maximized it on his screen so it was the only thing people saw.

“This was an easy one,” Omar started. “Let’s first talk about what we were trying to accomplish. We used page fourteen of the DevOps Automated Governance Reference Architecture paper as the basis for our approach. The risk we are addressing is unapproved changes, the control is peer review, the action that triggers the control to be enacted is a pull request merged into the main branch, and the actors are the code author and the code reviewer.”

Michelle had a big smile. She was amazed that Omar, a software engineer, was using a language that a Risk person would. She asked herself, Is this what they mean by shifting left with risk? Let risk management begin in the developer’s mind? 

“Before I get hands on, let’s talk about some things we learned,” Omar said. “Our source control tool has the ability to require a reviewer for a pull request. But this wasn’t enough, as Andrea and I discussed.”

Bill nodded his head, excited to hear that the engineering team was already seeking out advice and input from Audit this early in the process.

“Audit needs a record that the pull request happened. So we needed to find a way to generate that record. This hasn’t been done before. Other tools we use can issue a report, but the source control tool doesn’t do that. Also, before I forget, we decided that we only need to do this for the main branch, no other branches. That’s because code in the main branch goes into production. Other branches don’t go into production, and some branches may never get merged into main.”

Michelle interrupted. “Omar, before we go any further, can you explain to us all what pull requests, merges, branches, and the main branch are?”

“Are you freaking kidding me, Michelle, really?” Omar replied, in a half whisper, though everyone could hear him.

“We’ve failed in the past because the right people don’t understand the process. And the right people include Barry and Andrea and especially their teams back in Security and Audit,” Michelle said. “They need to understand how the sausage is made so they can help ensure we’re including their concerns correctly.”

Omar rolled his eyes as he accepted the task. “Okay, let me see if I can do this without explaining every aspect of source control and wasting everyone’s time. The purpose of source control is to track changes to code. The main branch is the golden copy of your code. When someone needs to update the code, they make a copy and add their changes. This copy is called a branch.

“For example, when we branch from the main branch, we give our branch the name of the ticket number for the feature we’re working on. So, let’s say we have a new feature for ticket 112233. We’ll create a branch with the name feature/112233. In branch feature/112233, we’ll make all of our code changes.

“When we’re done, we create a pull request. A pull request is pretty much as the name implies: it’s a request to pull the changes from another branch—in our case branch feature/112233—into the main branch. When the pull request is created, that’s when the code review happens. At that time, it’s just a request; nothing’s been merged into the main branch.”

“What do you mean by merge?” Andrea asked, tentatively raising her hand.

Omar answered, “Merging is the process of taking the code changes from branch feature/112233 and incorporating them into the main branch.”

“So merging is simply copying all the files from branch feature/112233 into the main branch?” Andrea responded.

“Um  .  .  . ” Omar turned to Michelle. “How technical do you want me to get here?”

“Just technical enough to help everyone understand.”

“Okay, you can think of it as copying. But what merging does is literally merge the content from the two branches, so only the changes in branch feature/112233 appear in the trunk. In fact, the reviewer can see the exact lines or file, or parts of lines or files, that are to be changed in the context of the whole.”

Andrea spoke up again, “Okay, let me restate to make sure I understand. When changes in some branch are ready to be brought into the main branch, and an engineer creates a pull request, this pull request kicks off a code review. That code review gives other developers an opportunity to see what is about to be changed and provide feedback on the changes. Then, I’m assuming when the code review is done, or done per the expectations of the review, the request is approved? And when approved, this is when the merge happens?”

“You got it!” Omar said.

Andrea replied, “I understand now. Tools aside, our auditing process is very similar. So, what we need is a) proof that a code review was made, b) who the code reviewers were, c) who the code authors were, and d) review activities such as any issues found in the review and evidence of those issues being closed or approved.”

“I was just about to get to that,” Omar said. “I have a demo source code repository here called Demo-Repo. There is one file called README.md. Inside this README file there is just one word, ‘hello.’ I want to change this, so I’m going to create a branch called change-1.” After some clicks and taps, a spot on the web browser changed from branch: main to branch: change-1.

Omar minimized the web browser and opened one of his coding tools. “Okay, I now have access to the new branch I created. I’m going to change the README,” Omar said. He put his cursor after the word hello and added a space, then typed world. The text now read hello world.

After some more clicking and reopening the web browser with the source control tool, Omar continued, “Alright, I now have the changes ready for a pull request.”

Omar hovered over a green button on the opposite side of the screen from branch: change-1 that said Pull Request. He clicked the green button and a new screen popped up. It had many things on it, but one field of text was particularly important.

“Here is where I’ll explain what my changes are and the reason for them.” Omar typed new requirement to add a subject of the hello. After that, he moused to the bottom right of the screen and pressed a red button that said Request.

“Dillon, can you review and approve that pull request when I give you the go?” Omar said, looking across the room. Dillon turned around and gave Omar a thumbs up. They watched on Dillon’s screen as he pulled up the pull request that highlighted the text world as the addition in hello world in the README.md file.

“Okay, we’ve simulated a change about to happen. When Dillon approves the pull request, the source control tool will merge it into main. Our continuous integration tool—we just call it the CI tool—will detect this merge and start doing other things. This part is where we spent a lot of time experimenting.”

“So far, there isn’t anything earth-shattering here,” Barry said.

“No, you’re right, but let me pull up the CI tool,” Omar replied, shuffling again through the multitude of web browser tabs.

Another window appeared on the screen with what looked like a process flow at the top. Most of the bubbles were gray, but the very first one was blue. Inside the bubble, it said Code Review Attestation.

“Here, where it says Code Review Attestation. This is what we did. Let me explain what’s happening under the hood. Sometimes these tech-demos suck, because the magic is happening behind the scenes,” Omar said.

He continued, “We wrote a small program we call Code Review Reporter. The purpose of Code Review Reporter is to create something similar to a report and make sure that those things Andrea talked about happen. When the CI detects a change, the CI tool is given a critical piece of info, something called a commit ID. This commit ID is a unique identifier that identifies the merge and the pull request. What Code Review Reporter does is send the commit ID to the APIs of the source control tool and asks two things: the names of all the code reviewers and all code authors. When we get that data back, we create a file that lists the names of the reviewers and names of the authors, counts the reviewers’ names, makes sure there is at least one reviewer other than the code authors, provides a number for the count of reviewers, and has a field we call attestation. In this field, if there is at least one reviewer that is not the author, we fill it in with PASS; if there’s less than one, then it says FAIL.”

Andrea, Bill, Barry, and Michelle looked super pleased.

“Okay, that’s slick,” Barry admitted.

Omar didn’t know what to do with a positive comment from Barry. He smiled and felt a burst of energy hit him like a shot of espresso.

“That’s not all,” Omar said. “Now, if the attestation says FAIL, then the Code Review Reporter tells the CI tool to stop processing. Then, nothing else happens in the pipeline and the code cannot go to production. Heck, it can’t even make it to its next steps. We call this breaking the build.”

Everyone at the table was looking at each other. They were attempting to assess what the others thought of what Omar and the team had delivered.

“Something is missing here, but I can’t put my finger on it,” Andrea said.

Omar looked surprised. The excitement he had been enjoying suddenly died.

“What? What could be missing? I literally just accomplished building a tool to fulfill the first actionable item. Nothing can go to production without having at least one reviewer,” Omar said with agitation, gesturing wildly at the screen as though no one had just seen the brilliance he had shown them.

“That is your segregation of duties right there. No single developer can make changes to code and send it to production without a second set of eyes reviewing it.” Michelle looked at Andrea.

“Yes, on the surface this works, but something is missing. I just can’t figure out what it is,” Andrea replied.

Omar let out an exasperated sigh and crossed his arms.

Michelle said, “Andrea, let’s set up some time to talk more about it. Could you take the rest of the day to compile your thoughts, and we can cover it on Monday?”

“Yes, I can do that,” responded Andrea.

“Add me to that list as well,” Barry requested.

Omar’s badges briefly wow the suits, but Andrea spies an evidentiary gap while Barry blasts Omar’s oversight! Smooth sailing remains distant as the crew realizes silo perspectives still clash! With foreign concepts hanging over Team Kraken’s vision, can true empathy and common purpose prevail? Join us next time for the continuation of the story. Or, go to your favorite book retailer and pick up a copy of Investments Unlimited today.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•