Attack of the Supply Chains – Investments Unlimited Series: Chapter 9

Welcome to the ninth installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

In our last installment, complications mounted. Now, with the regulatory reckoning nearing and prototypes piled high, a supply chain cyberattack pulls Michelle’s engineers off-track again! In crisis talks, auditor Lucy prompts provocative ideas around “dependency” risks, but will impatient Omar or pragmatic Michelle connect these dots in time?

Thursday, September 1st

The next month flew by quickly. With the post-lift-and-shift Omega project in the rearview mirror and a renewed energy from their recent successes, Team Kraken was making significant progress with Turbo Eureka. The turbo of Turbo Eureka was really working! They had implemented more quality gates and really felt like they were on a roll.

Michelle and Bill had just finished giving another demo session to a broader audience. Other people within IUI were becoming interested in what the team was doing. They were even using Turbo Eureka to automate the governance for all Turbo Eureka software they developed.

Michelle sat down at her desk and looked at her calendar. It had now been six months since they had first received the MRIA. She opened her laptop and navigated to the MRIA Outline document in the MRIA Madness folder. She had been keeping track at a high level of Team Kraken’s accomplishments. 

Actionable Items: Based upon the MRAs issued, the following items should be addressed with formal standardized approaches:

Goal: Define a minimally acceptable release approach

Objectives:

• DONE: Enforce peer reviews of code that is pushed to a production environment.
• Identify and enforce minimum quality gates.
  • DONE – Unit Tests
  • DONE – Source Code Quality Analysis
  • DONE – Static Application Security Test
  • In Progress – Software Composition Analysis
• Backlogged: Remove all elevated access to all production environments for everyone.

The team had even been able to add some nice features to the Git repo. They continued using the open-source project that created software for badges that were color coded to make it easy to visually understand the status and quality of the software in a Git repo.

Michelle pulled up Team Kraken’s Git repo on her computer. In the middle of the screen, offset to the left a bit, she saw the badges.

This repo had a badge with the status of every quality gate, showing whether the most recent quality gate passed or failed. The first badge simply showed the version of the software. The left part of the badge had a gray background with white writing that read Version. The right-hand side of the badge had a baby blue background color with gray text that said 0.0.4.

Below the version badge, there was a badge that read Unit Test. The right-hand side of the badge had a green background with the word PASS in it followed by a check mark. There were a few more subsequent badges that showed green, just like the unit test badge. Then there was one that read Software Composition Analysis. The right-hand side of the badge was red with the word FAILED inside of it and a large X.

“Omar, have a second?” Michelle hollered across the floor.

Omar got up from his desk and walked over to Michelle, looking at her screen. “What’s up?”

“Why is the software composition failing for this repo?”

“Let’s see  .  .  .  what’s the latest commit number for main?”

Michelle clicked around on the web browser and highlighted some text. “The most recent commit to main is 9349c9b.”

“Okay, hold on to that number. Open another browser tab and go to https://
attestations.investmentsunlimitedbank.com.”

Michelle typed the address in.

“Okay, paste the commit number into that box,” Omar said, pointing to a box labeled Enter Commit Number on her screen.

Michelle pasted the number into the box and clicked Find Attestation. A table popped up. It only had one row. She clicked View Attestation and the page refreshed, rendering all of the different attestation files. She scrolled down until she saw a title with Software Composition Analysis inside of it.

There was a lot of information in this part of the file. She saw recognizable sections that were named High, Medium, and Low.

Omar pointed at these sections. “Those are lists of common vulnerabilities and exposures—CVE for short. See, right there,” Omar said as he read the screen. “Look at this Critical: CVE-2021-44229. If you remember, by policy, there shouldn’t be any CVEs in the critical or high category.”

“Do you know what that CVE is for?” Michelle asked.

“No, we’d have to look it up. But on the bright side, it looks like Turbo Eureka is working!”

“I guess it is,” Michelle replied. “Okay, thanks for the info.”

Omar turned around and went back to his desk.

Michelle scrolled through the attestations website a bit longer. She was impressed with all the information they were collecting and the controls they were implementing.

A few hours later, Omar came up to Michelle. “I just saw in my Twitter feed that there’s a new Java vulnerability causing problems. Do you remember which CVE you saw this morning?”

“Who posted on Twitter?” Michelle asked.

“Look at this. It looks like some security firm,” Omar replied, showing her his phone. “But it’s really started to blow up on Twitter. Can you pull up the one that you asked me about this morning?”

“Yeah, one sec.” Michelle spun around to her laptop and pulled up the attestations website again.

“Oh, yeah! It’s the same one!” Michelle practically shouted.

“Have you heard anything from Barry?” Omar asked.

“Not a thing. Let me reach out and find out?” Michelle picked up her phone and started typing.

Hey Barry, have you heard anything about this new critical CVE thing? Some Java vulnerability that’s blowing up on Twitter?

Omar went back to his desk. 

Michelle waited for Barry’s response for some time, but eventually evening rolled around and she headed home. Maybe she’d find Barry tomorrow and ask him about it.

“Mmm. I think the fruity flakes are my favorite,” Michelle said.

“No way, Mom. The peanut butter and chocolate puffs has that beat any day.” One of Michelle’s sons was spooning ice cream into his mouth as fast as he could.

“I’d have to agree,” her wife said.

Michelle was with her twins and wife at Revolution Ice Cream. Her twins had melted ice cream running down their faces. She couldn’t help but notice most of it was on their shirts.

Michelle’s phone pinged and vibrated. “Nope,” she said while fishing for her phone. “Fruity flakes all the way.” She found her phone at the bottom of her purse and looked at the text.

Michelle, can you call me? it read. It was from Barry. It wasn’t common for him to text Michelle, especially after hours. Barry was an email person. He hadn’t even warmed up to chat. She got a bit tense.

Out with the family. Wait until tomorrow? Michelle texted back. She knew what the answer would be, but this was her polite way of telling people to bug off.

Me too. This can’t wait, Barry’s short reply said.

“Oh no, not another one,” Michelle muttered. She started to get nightmares from the lift-and-shift Omega fiasco not so long ago. Was that acting up again? She and the team had done a significant refactor to the application. Then she remembered the CVE she had asked Barry about earlier that day.

Her wife looked over. “Everything okay?”

“Can you watch the kids for a second? I need to make a call. Something’s up at work, and I need to check in on it.”

“Sure.”

“Thanks,” Michelle said as she scooted out the door.

“Michelle, we’re up shit creek without a paddle,” Barry said when he answered Michelle’s call. “Our Network Operations Center picked up some suspicious network traffic earlier today. Lucy called me a little earlier saying she got a text from the NOC to join a call. And I’m just now seeing your text. What’s going on? I’m worried it’s related to the chatter we’re hearing too.”

Michelle replied, “I was thinking of asking you that. You’re the security guy. All I know is  .  .  . ”

Barry didn’t let her finish her sentence. “Michelle, I’ll have to call you back. We may have a problem here. Tim is pinging me. Bye.”

Michelle’s heart sunk deep into her stomach, so much so that her ice cream almost made an encore.

Michelle scooted back inside. “Grab some to-go lids for the kids. We need to cut this short,” she said to her wife. She stood up and reached for the twins’ ice cream cups. “Can you drive? I may need to take a few calls on the way,” Michelle added.

Five minutes later, Michelle’s phone rang. It was Barry again. 

“What’s up?” she asked.

“It’s bad. That network chatter is related to the critical CVE that you texted me about. Tim said the NOC is getting flooded with alerts. I have friends texting me from other companies. I’m about to hop on a call with our security consultants at AlertFirst to see what they know.” Barry paused.

“What do you want me to do?” Michelle asked, hoping she wouldn’t have to get on her laptop tonight.

“Stand by? After this call with our friends at AlertFirst I’ll know more,” Barry said.

Over the next hour, Barry kept Michelle up to speed via a flurry of texts on their inter-office communication channel. According to AlertFirst, a security consultancy IUI hired for computer forensics and penetration testing among other things, what was happening at IUI was not an isolated incident. In fact, it was quickly turning into a global firestorm.

Soon, news of the vulnerability was everywhere, even trending on social media. Companies around the world were scrambling. Technology teams from every sector were all focused on this one issue. It was as if someone had hit an international “pause” button on everything else in the tech space. 

As the night rolled on, it turned out the vulnerability was pervasive yet overwhelmingly elusive. Every time the IUI NOC team thought they had addressed every vulnerable application, a new one would surface or a third-party software would publish a new patch. All other work at IUI was suspended while they directed their efforts to eradicating the vulnerability. 

Friday, September 2nd

“Michelle, I think our security consultants found something interesting,” Lucy said. Lucy was on IUI’s security team and was an expert on IUI’s central logging platform, which was used to collect any data that was created by applications and hardware under IUIs control.

She was grinning from ear to ear. It was clear that Lucy was enjoying the challenge, unlike everyone else in the NOC and most of IUI’s technology team, who looked like they had just survived an harried expedition across a dense, dangerous jungle. 

“What’s up,” Michelle asked, worried it was going to be more bad news.

“I was on the phone until late last night with our AlertFirst consultants,” Lucy answered, looked excited rather than bothered. “The NOC has kinda stopped the bleeding for now, but the patient’s still in serious condition. This is a critical vulnerability, as you know, and we still haven’t fixed it or found a cure yet. No one has. We’re all just on damage control.”

Lucy continued, “Anyways, what I wanted to talk to you about is this, it seems this vulnerability is being caused by a specific Java library. Our AlertFirst consultants believe this as well, and we’re hoping you can help us confirm. Can you help us check our applications for this dependency? ”

As Lucy was trying to pull up her notes from last night, Barry and Andrea came walking over to listen in on Lucy and Michelle’s conversation. Everyone at IUI was deeply involved in the incident. With the MRIA still over their heads, they couldn’t risk a breach. 

Michelle replied, “Hmm, that’s a pretty common dependency. I wish I had better news, Lucy, but it might take some time to find them all. Omar, can you cross-check our applications for this dependency?”

Omar began pulling up Git repositories and everyone stood around awkwardly, wishing they had some way to help. But this was just going to take time.

“So, while we wait, I’d love to understand this better. What is a ‘dependency?’” Andrea asked.

Michelle turned to Andrea, “It’s an open-source library that we use in our applications.”

“An open-source library?” 

“Oh, well, software engineering, at its heart, is simply writing code. Like instructions to bake a cake—well, a highly complex cake,” Michelle said, tapping her foot impatiently on the floor while staring over Omar’s shoulder. 

“But there are lots of ways to bake a cake, and there are lots of steps,” she continued. Andrea listened intently. “While we may write a lot of code ourselves, kind of like writing our own recipes, we save a lot of time and effort just grabbing bits from others. You know, why reinvent the wheel when the code is already out there in the open? You can use this ‘open’ software that is stored in public code repositories or you can use bundles of this software that we call ‘libraries.’ Our code depends on these libraries. These are our applications’ dependencies.”

“Oh, so you’re using pieces of someone else’s recipe to bake your own cake?”

“Yep,” Michelle replied.

Everyone continued to watch as Omar went through Git repositories on his laptop. Every time he found an app using the dependency, he put a check mark by the app’s name on a sheet of paper, a list that was quickly growing longer and longer.

Michelle asked, “Omar, can’t you look at just the Java repositories?”

“Wish I could tell just by their names,” Omar replied.

“So, when you open a repository and find it’s using Java, can you tell if it is using the dependency?” Michelle asked again.

“I think I can tell if our code is directly using this dependency. But I couldn’t tell easily if it’s a transitive dependency,” Omar replied.

“Transitive dependency?” Andrea asked.

“Basically it’s like the recipe we borrowed had already borrowed a recipe, or bits of the recipe, from someone else,” Michelle said, a little frustrated.

Andrea gave her a quizzical look.

“Let’s say we want vanilla icing for our cake, but we don’t want to make it from scratch. So we get the premade stuff from the store. But the company that made that icing didn’t want to make their own vanilla flavoring from scratch; they bought it from somewhere else. That’s a transitive dependency.”

“So if the vanilla used in the icing is bad, my cake will be bad, and I have no control over it?” Andrea asked.

Lucy burst out, “Yup. It’s a software supply chain problem. Just like if your icing company bought vanilla flavoring that had been tainted or whatever.” Lucy looked like she was going to explode with excitement. She was the only one. Everyone else looked like they could use about a week’s worth of vacation time on a secluded beach.

“You see, a supply chain is the entire production flow. It’s everything involved to deliver a product: the people, the materials, and the activities that produce a product like a vanilla frosted cake. The only difference here is that we’re not talking about a cake, we’re talking about software.”

Andrea got excited for a minute. “This is fascinating.”

“Really?” Omar quipped.

Lucy continued, ignoring Omar and engaging directly with Andrea. “The problem is, when I go to a store to buy a cake or vanilla frosting, I’m not likely to ask who produced the vanilla flavoring. And it might not even be something the frosting company or grocery store readily knows!”

“No,” responded Bill, “I wouldn’t even think to ask. I trust the store I’m buying from, therefore I trust they wouldn’t purchase any bad vanilla or software with bad dependencies.”

“That’s a good point,” Lucy said. “Because you trust the store, you’re implicitly trusting the full supply chain of the cake. You’re trusting all the people, activities, and resources involved in that supply chain. ”

“We probably do the same thing in software. If we trust the software vendor, we trust the whole supply chain of the software,” Bill commented.

“I may be changing the topic a bit,” Andrea joined back in the discussion. “But I find developers trust open-source software more than vendor software.”

“Isn’t that natural? I mean, everyone can see the code!” Omar added, still slowly adding apps to his list.

Lucy responded with a question: “Are you sure about that? What do the rest of you think?”

Everyone’s eyes darted around the semicircle, looking at each other. Michelle had this sense that Lucy’s question was loaded, but she couldn’t think of why.

“That sounds like a trick question,” Andrea responded.

“Omar’s right,” Bill blurted out. “Anyone can see the open-source code, anyone can review it, test it. On the other hand, you’re relying on a company that created the closed source. You don’t know what’s in their code.”

“I agree,” Andrea said out loud.

“Me too,” said Michelle.

“Me three,” said Omar.

“Well, I think you’re wrong. Open source is no better than closed source, nor is it any worse. You assumed that because it was open, it was being reviewed by many other people. But that’s just an assumption. You don’t know it to be true. This is an example of something called ‘diffusion of responsibility.’” Lucy did sound very academic.

“Wait,” Omar paused his list making and spun around in his chair. “You’re telling me that even though open source is open, it’s no more likely to be safer than closed source?” 

“That’s exactly what I’m saying. That’s where diffusion of responsibility comes in. Diffusion of responsibility refers to a situation where as the number of bystanders increases, the personal responsibility that an individual bystander feels decreases. As a consequence, so does their individual tendency to help. So, for an open-source project, someone using that project assumes that the project’s team and other users are ensuring some level of quality. If all users of that project feel that way, effectively no one is actually reviewing anything,” Lucy said.

“Lucy, it may be the hangryness or the lack of sleep, but I don’t see how this is getting back to that dependency issue we’re having,” Michelle responded heatedly.

“Yes, the dependency. Sorry for the diversion, it’s just that this is so interesting.”

Omar gave Michelle a pointed look before spinning back around to his laptop. Obviously he didn’t think so either.

“We think the dependency that’s causing this problem is an open-source project, based on talks with our security firm. But we don’t know yet if it’s a software supply chain attack,” Lucy explained.

“How can someone attack a supply chain?” Andrea asked.

“Issues caused by someone in the supply chain can be unintentional or nefarious,” Lucy said. “For example, that vanilla we keep talking about. If the vanilla manufacturer didn’t properly clean their equipment or the production line had a bacteria issue, and then all of the cakes using that manufacturer’s vanilla could make people sick. This could affect people throughout the world, depending on who purchases that manufacturer’s vanilla and where they are. The same is true if there was a disgruntled employee. Someone with nefarious intentions could poison the vanilla, causing the same issues. The outcome is still a lot of sick people,” explained Lucy.

“We think it’s the same with our software supply chain, like the dependency in question right now. It could be just a coding error or a bug that wasn’t caught before release. But if someone nefariously introduced some malicious code in the project, then we’d have a software supply chain attack,” said Lucy.

“Ah, I get it now,” Andrea said. “Somehow there is a flaw with this dependency. This flaw has gone overlooked. Since everyone just trusted things were good, without validating, this flaw crept its way into our software. It reminds me of something my dad used to say: ‘The road to hell is paved with good intentions.’”

“The only thing I don’t understand is how could we have prevented this?” Omar said, spinning around once more and handing Lucy the very long list of affected apps. “What could we have done better? Is this something we can check for when we build our own software?” 

“Perfect question!” Lucy replied. “And one I don’t have an answer to.”

“Hey, gang!” someone shouted across the room. “The bosses got catering for a late lunch. It’s all set up in the NOC, so go get yourself a bite.”

“Let’s get Michelle to the front of the line. She’s slowly transforming into a hangry monster,” Omar said, only half joking.

Michelle quickly threw him a mean glance.

Eureka! Together Michelle and Lucy crystallized the power of SBOMS to trace every software ingredient to its source! With Omar’s graph prototype, a new possibility arises to track assets enterprise-wide. Could this craft future-proof governance and help developers monitor risks proactively? Signs now suggest the Kraken’s on the cusp of a potential breakthrough! But bumpy political battles still surely await on the road toward reform! Join us next time for the continuation of the story. Or, go to your favorite book retailer and pick up a copy of Investments Unlimited today.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•