SBOMs and More – Investments Unlimited Series: Chapter 10

Welcome to the tenth installment of IT Revolution’s series based on the book Investments Unlimited: A Novel about DevOps, Security, Audit Compliance, and Thriving in the Digital Age, written by Helen Beal, Bill Bensing, Jason Cox, Michael Edenzon, Tapabrata Pal, Caleb Queern, John Rzeszotarski, Andres Vega, and John Willis.

In our last installment, supply chain woes threatened to derail everything. Now, can Michelle finally shift focus back to her critical MRIA modernization mission? Hold-ups and headaches have dogged her band of pioneering engineers, who failed to meet specifications at every turn! Now, yet ANOTHER roadblock looms, with some applications still failing to sync up with the new Turbo Eureka system! Have deadlines and executive doubts bred an unsolvable Catch-22? Or might Michelle’s mentality somehow still rally the troops toward governance glory?

Wednesday, September 21st

It took several weeks for the IUI team to get the supply chain problem under control. They assumed it would be an easy fix once they identified all the affected software. They were wrong. 

At times, it seemed like a clear journey, although they learned the hard lesson of mistaking a clear journey for a short distance. The team had to work on it for weeks, including weekends, pulling focus away from Turbo Eureka. And they weren’t the only ones. It seemed every tech organization in the world had spent the last few weeks painstakingly searching for every affected app. Michele had never experienced anything like it.  

Inside IUI, calls for an outside engineering firm to come in and help with Turbo Eureka became louder. No matter what Team Kraken did, the naysayers were coming out of the woodwork, like cicadas come out, en masse, every thirteen years or so. The chastising, blaming, and sometimes name-calling began to take an emotional toll on everyone on the team, even Michelle.

She felt like she and her team had fallen from some type of grace. All the hard work and progress they had been making with the automated governance tools were suddenly forgotten as their deadline was fast approaching. They had mitigated the latest fire by patching all instances of the Java dependency, but they were no closer to solving how to avoid software supply chain vulnerabilities in the future. It started to get so bad that Michelle kept getting wind of engineers putting their resumes out, looking for other positions.

It was mid-afternoon on a Wednesday when Michelle saw Lucy walk into the dungeon. Her heart raced for a minute, worried there was a new vulnerability that was going to cause another massive delay. But Lucy was all smiles.

Bill, Michelle, Barry, Andrea, and Omar were standing around two high-top tables reviewing actions for their next iteration of the Turbo Eureka automated governance tool. As she approached, Lucy jokingly said, “If only these tabletops could talk! Looks like the brain trust is hard at work.”

“Loathing is more like it,” Michelle replied. “We completed automating more gates for our deployment process, but we’re still not feeling hot about the product. This constant talk of bringing in consultants is driving me insane.”

“Yes, me too,” replied Bill. “It’s as if they’re asking on repeat as a way to wear me out. All with the hopes that one day my response will be ‘Yes, they’ll be here tomorrow.’ And then, magically, all our problems will vanish with a flick of their magical consultant wands.”

“I’ve been doing some reading on software supply chain analysis tools,” Lucy said. “I’m almost convinced that something more can be done as part of your automated governance that would help in future supply chain-related issues. Have you thought about generating a valid SBOM as a gate for your continuous integration?” Lucy asked.

“An S bomb? Is that like an F bomb?” Omar asked.

“It stands for software bill of materials. SBOM.”

“No, we haven’t,” Michelle replied. “But I’ve begun to hear that term more and more.”

“Well, enlighten me, please,” Omar said.

Lucy responded, “It’s simply a list of all the components you use to build your software, like the one that caused all our issues not too long ago.”

“Don’t remind me,” Michelle groaned.

“Really, that’s all it is? Heck, we already have that. It’s called a POM file in Java if you build with Maven,” Omar replied without caring to explain further.

“Mmm, yes and no  .  .  . ” Lucy equivocated.

“What?! A POM file is exactly the same thing. How am I wrong?” Omar questioned, as if Lucy was on trial.

“Well, the idea has been around for a while, although its initial incarnations were simply as a way to track licenses for software components,” Lucy continued without directly answering Omar’s question. “Recently, with the rise in exploits of supply chain vulnerability issues, the concept of a software bill of materials is a thing as well as a rally cry to people across industry. It’s an attempt to raise awareness about the quality of software, and most of the zeal has been aimed at open-source software. But honestly, if you ask me, I think too many people are trying to use it as a silver bullet.

“According to the US Department of Commerce, an SBOM is a formal, machine-readable inventory of software components and dependencies, information about those components, and their hierarchical relationships. On May 12th, 2021, the President of the United States issued an executive order on improving the nation’s cyber security. The executive order specifically calls out commercial software providers to provide an SBOM, so that federal or regulated entities are able to quickly identify vulnerabilities or threats from the dependencies used in commercial software. It also enables companies using commercial software to attest software before it’s installed.”

“Okay, but you still haven’t told me why I’m wrong,” Omar rebutted.

“An SBOM is an extensive list of all software components and dependencies. Omar, from your perspective, for someone who is writing an application, it would be not only the list of dependencies you primarily depend upon, like the ones in your POM file. It would be a list of all the transitive dependencies, including extra information such as version numbers and where that dependency came from. It’s your full dependency tree, a full family tree of what uses what, all the way down to the roots.”

“Oh  .  .  .  that’s a bit more complicated,” Omar said in surrender.

Lucy continued, “It doesn’t end there. I think the SBOM should include the machine the software is running on, the version of the operating system running on that machine, where that machine is located, and so on, even down to the other software running on that machine alongside yours.”

“Isn’t that what our Configuration Management Database does?” Bill asked.

“The CMDB?!” Lucy said with a full belly laugh. “Setting my true feelings aside for a CMDB, Bill, sort of. The idea of the CMDB is to logically lay out how software and hardware assets interact with each other. The problem with a CMDB is it’s generally out of date the moment it’s updated. I’m not sure I’ve ever been in a company where you could truly trust it. Most of the time, it was the best guess at what our technology landscape looked like, but it was never 100% correct. And I’m not talking about a little off—it’s generally way off.”

Bill responded, “Okay, how the CMDB is maintained may be an issue, but in concept it’s similar, right?”

“Yes, I’ll concede that.”

“So, how do we go about generating an SBOM?” Omar asked. “That’s a ton of information that might be hard to get. And who’s going to do it? Who has time?”

Lucy replied, “Does your software composition analysis tool not do this for you? They may not explicitly call it an SBOM; they’ll most likely call it a dependency tree or a dependency analysis. Usually the tool will check your tree of dependencies to see if there are any CVEs for that specific dependency and version.”

“Actually, it does. We’re using this report for the SCA quality gate of our automated governance right now,” Michelle replied.

“That’s great to hear. There are industry agreed-upon formats like CycloneDX or SPDX. It wouldn’t be too hard to create an SBOM in one of those formats from that data,” Lucy replied.

“Hold up, this isn’t making any sense. Lucy, you’re telling us to do something we’re already doing, which is checking for vulnerabilities at build time with the SCA tool?” Omar interjected.

“Close. You can’t just scan a dependency once. At the time you scanned the dependency there may have been no vulnerabilities. That’s simply because no one was aware. A new vulnerability may be published any time, and unless you build your application at least once after that you won’t know about the vulnerability. The problem with how lots of folks use SCA is that the scan happens only during a build that is triggered by a code change,” Lucy replied.

“Ohhhh,” Omar said as his eyes lit up. “I get it! Old software that we’re running on our machines but not developing anymore might have vulnerabilities. And we wouldn’t know because we haven’t scanned them recently!”

Omar paused, reflected on his own observation, and involuntarily shuddered. Snapping out of it, he asked, “Okay, but how does this all tie back to the SBOM again?”

“Well, the SBOM is a list of all dependencies, as we discussed before. So instead of always rebuilding software just to scan, you can simply scan the SBOMs for all newly published vulnerabilities and know if there are impacts. You can also use this to proactively upgrade old dependencies or replace vulnerable ones used in multiple locations or products,” Lucy stated.

“Okay, so an SBOM is like a map. When something new happens to a dependency, a new CVE for example, we can easily locate it on the map and see what roads lead to and from it. Then, with this information, we can take some action to address it,” Omar said.

“Yes,” Lucy replied. “We like to refer to this as detective versus preventative controls. Preventative controls are things we do to prevent something bad from moving forward. These are what our build-time quality gates are responsible for. Then there are detective controls. Detective controls are things, like an SBOM, we can do to validate a system is still in compliance as the things around the system change, like the dependencies. Your running software itself will not change while in production, but an exploit could be identified. With checking for compliance at rest and at runtime, we can detect these new situations.”

“That gives me an idea,” Omar said. “I’ve been playing with this new graph database. I bet I could create something that stores the dependency list from the SCA tool and give the ability for us to search for a dependency. That way, if something like this happens again, we can just query for the dependency and see what the affected systems are.”

“Oh, I like this,” Michelle said, her eyes widening in excitement. “I’m always keeping on the engineers to update their latest dependencies. I can use this to identify who needs to keep their stuff up to date. I bet we can also apply some analytics to the SBOM data to finally help us rationalize what technologies our teams are using to build our software and be more intentional about what frameworks we’re placing our bets on!”

“Maybe this isn’t such a good idea,” Omar responded, tongue-in-cheek. “But seriously, give me about a week and I can have an MVP put together for us.”

Friday, September 30th

In usual fashion, what was supposed to be a week turned into nearly two weeks.

It was Friday morning, and Michelle was huddled over Omar’s desk. He had finished the prototype late last night and wanted Michelle to be the first set of eyes on it.

“Look up CVE-2020-17530 and see what comes back,” Michelle asked him.

“Oh, wow—unless my prototype is buggy, that’s a lot of results,” Omar replied.

“No, that doesn’t surprise me. I’ll fill you in later. Let’s demo this. I’ll grab the others,” Michelle said as she ran off.

About thirty minutes later they were all gathered around the high-tops in the dungeon. Bill was the last to arrive.

“Hello everyone,” Michelle said.

“Unless Omar is telling fairy tales, it sounds like you all have made some good progress,” Bill said.

“I think we have, but we’ll let you be the judge. Check out our SBOM approach,” Michelle responded, turning to Omar.

Omar proceeded to walk everyone through their continuous integration process so far. He rushed through gates such as code review and unit testing. He hurried to the software composition analysis gate. Omar added the SBOM procedure after the SCA gate. When the SCA gate passed, this subsequent procedure took the dependency tree and created a new record of it in Omar’s graph database. Once this was done, the rest of the continuous integration process continued as normal.

Omar turned back to the rest of the group and explained that with the interface he provided, when a new CVE is issued, anyone can simply enter the information about the dependency inquisition, then click Search. To demonstrate it, he pulled up a new web browser on his computer and went to the new demo SBOM Search site that he created and entered CVE-2020-17530. Numerous results came back.

“Here are the applications that are impacted by this CVE. You’re seeing not only what applications use this as a primary dependency but also those that have this as a transitive dependency.”

Bill looked at Barry. “What does this mean for Security?”

“It’s risk mitigation. Think of it as continuous monitoring, just like we do in the NOC. What Omar showed is how we can continuously search through the bits and pieces that make up our applications to identify a bad part, then see what other parts of the system they affect. It reduces our mean time to act in the face of a vulnerability or exploit caused by a software dependency.”

“It’s really the epitome of real-time supply chain compliance,” Andrea added. “For this type of situation, we can assess, at a moment’s notice, what our risk exposure is to any identified software supply chain vulnerability. Whereas before, we’d just repeat the mistakes we made over a month ago. Now, we can identify affected systems within seconds and enact control procedures to mitigate any risk. In addition to this, we can now get proactive with our risk management. As each of these components has new versions available, we can use this to identify all the systems that need to be updated to the newest version. Keeping our software dependencies updated with the latest version releases is a leading way to mitigate risk. We can now enforce compliance at build time, rest time, and run time.”

Barry turned to Omar, “Is this hooked up to our CMDB?”

Omar, with a flushed face, responded, “No, it’s not. That was not the best approach.”

Barry seemed to get visibly flustered. “Why not? It seems that this is exactly why we have a configuration management database!”

“Searchability is key,” Omar replied. “I don’t want to go into too much detail, but our CMDB has limited search capabilities and can’t be used in this problem. I used sophisticated graph search algorithms for this. ”

“Kid, I did my computer science degree forty years ago. I try to stay current but I’m not following the graph search algorithm reasoning. You’re saying we can’t do that with our CMDB?” Barry replied.

“Okay, sorry, ” Omar said in an apologetic tone. “But yes, our CMDB can’t handle this. With a depth-first search, we can take a piece of software we wrote and ask, ‘Does this software have a dependency upon X?’ The depth-first search will traverse all the dependencies of the dependencies of the dependencies to find out if the dependency in question is being used, either as a primary or a transitive dependency. It’s the same type of problem you’d come across if you were navigating a maze. Depth-first helps navigate a maze-like question. Does that make sense?”

“Yes, it’s starting to come back now,” Barry replied.

“Well then,” Omar continued, “with a breadth-first search, we can ask, ‘What software uses dependency X?’ It’s similar to what web crawlers do for search engines or when you ask your GPS ‘Where are all restaurants located around me?’ Each of these methods provide us a way to analyze our software dependency relationships across differing software bills of materials created for our systems. Our CMDB doesn’t do this.”

Barry didn’t know what to say. It was as if Omar politely called his kid ugly, and Barry wanted to agree that his kid was ugly.

“Okay. This is great everyone. What we’ve just seen here is simply impressive. Omar, I understand it’s just a prototype, but if we can get this into production as soon as possible, we have something really special here.” Bill sounded very happy.

“Agreed,” Andrea said.

“We need to update the external audit team on our progress, and it’s time to give the regulators another update. Omar, Michelle, I want you both ready to give a full presentation to stakeholders next week. Andrea, can you brief Jada and get her up to speed?”

“Will do,” Andrea responded, as they all stood up to leave.

Legacy loopholes and pipeline discrepancies threaten Michelle’s valiant home-stretch dash toward regulatory redemption! Pressure rises exponentially with higher-ups demanding all-or-nothing results! Heads spin scrambling three-letter acronyms, obscure system intricacies, and contradictory data sources! Is a through line toward continuous compliance now finally in sight, or have too many cooks and executive egos hopelessly spoiled the broth that is automated governance reform? Join us next time for the continuation of the story. Or, go to your favorite book retailer and pick up a copy of Investments Unlimited today.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
• 

Helen Beal

Coauthor of Investments Unlimited.

Follow Helen on Social Media

• 
• 

Bill Bensing

Bill Bensing tranforms Shadow IT into legitimate software development organizations. Bill's recent thought-leadership is proving software devliery velocity and highly secure and compliant software are not mutally exclusive. He lives in Tampa Bay, FL, area.

Follow Bill on Social Media

• 
• 

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Michael Edenzon

Michael Edenzon is a senior IT leader and engineer that modernizes and disrupts the technical landscape for highly-regulated organizations. Michael provides technical design, decisioning, and solutioning across complex verticals and leverages continuous learning practices to drive organizational change. He is a fervent advocate for the developer experience and believes that enablement-focused automation is the key to building compliant software at scale.

Follow Michael on Social Media

• 
•