Bank on Open Source for DevOps Success

Open source has been one of the core elements in Capital One’s digital transformation journey over the past 6 years. In this presentation, Topo Pal and Jamie Specter, will explain why open source is a necessary part of the journey, the pain points they have experienced along the way, and how their engineers partner with Legal and Security to overcome those challenges.

Topo Pal

My name is Topo Pal. I’m Senior Director and Senior Distinguished Engineer at Capital One. Basically, I’m a developer, a DevOps evangelist. I’m also a member of our Open Source steering committee, and I’m the creator and co-contributor of our flagship DevOps dashboard project, open source, called Hygieia.

Jamie Specter

My name is Jamie Specter.

Part of my role at Capital One is Open Source counsel. What that means is I get to partner with Topo, our Open Source program, and our development community on all things open source. That includes use, contributions, and external projects.

I’ve been with Capital One now for six years, and I’ve had the incredible opportunity to partner with people all across the enterprise.

Capital One

Many of you may know Capital One as a credit card company. We’re one of the largest credit card companies in the US, with over 70 million accounts. You may also know that Capital One is also one of the largest banks in the US.

What you may not know is that Capital One is a founder-led technology company. In fact, we’re about to celebrate our 25th anniversary from going public next year. Our youngest competitor is 108 years old. In that sense, we are a startup, and because we’re founder-led, we have that entrepreneurial mindset.

From the day we were founded, Capital One has always tried to be disruptive. It’s part of our DNA. Open source and DevOps has contributed to that success, despite the regulated industry in which we operate.

Topo Pal

We are different. We have different DNA because we build our own software, and we build it on public cloud. We deploy our software using microservices architecture. We are also open source first, as a company. We strongly believe in continuous delivery. Our belief is that if you are not doing continuous delivery in a regulated industry, you are not “compliant enough.”

About six years ago, we were mostly outsourced, 100% waterfall, with manual processes all over the place. We were slow, as you can imagine. Only commercial software was allowed, and we also said no to open source.

The journey to today took us from mostly outsource to mostly insourced, and I think this is really important for our digital transformation. Vertical silos like Ops, Dev, QA, etc. to a product-centric approach where the product team includes everybody that needs to be there to actually do everything related to that product.

And as I said, from Devs, Ops, QA, RN, we had these different roles. And we went from ‘You’re developers, you’re operations,’ to ‘we are all engineers.’ Developers write some code, operations write some code, infrastructure has code, QA engineers write some code, even release managers have become release engineers.

What We’ve Shared

• In 2014, I talked about building out our automation steps.
• In 2015, I talked about how we tried to scale DevOps, open source, and cloud innovation across the enterprise.
• In 2016, we talked about who we are as we concentrating on measuring, improving, and maturing our developed sources.
• In 2017, I talked about better governance via continuous delivery. That was very important to us because we are regulated. We cannot do things that are not allowed, so we have to be very careful about our journey, and we believe that continuous delivery is the way to go.

For 2018, when I was thinking about the topic that I wanted to share about, I found The State of DevOps Report from DORA. For the first time, they have included open source in the study. And I thought that that is the natural progression, to talk about open source and DevOps.

Say ‘No’ to Open Source

As I said, about six years ago, we said no to open source, and there were many other enterprises like us, big and small. They all said no to open source, while still coding Java, and using your Spring Framework, or coding Eclipse IDE, and using Apache Ant. I don’t know if that makes sense. What it means is that everybody used open source, but they were in denial about it.

Back in 2012, when we first started our continuous integration journey, we started with this basic toolset. I brought in Hudson because Jenkins was just born that time. We brought in Maven, SonarQube, Nexus, and then, it blew up. It didn’t work. And that is because the commercial source code software didn’t have enough capability.

We went back to the vendor, who said, “Oh, the version that you have doesn’t support that. You need an upgrade, and that costs money.” And we said, “Okay, we are not doing that. We are bringing in Subversion.” Today we don’t use Subversion, but back in the day, we started using it and on day one, it worked. Magic.

Commercial CSM to Subversion

That was a big deal for us, going from commercial software to commercial source control, to Subversion, because it was a free tool. Are you going to use a free tool for our critical business needs, will it be the thing that holds all of our commercial software? That’s risky. What if it breaks? Who do you call? I don’t know. Are we giving our IP away, because we’d be putting source code in that open source code? Does that mean that we have to open source our code, too? What if somebody steals our code?

We went through all these problems, and we talked about this. Finally, we came to a realization that to get Subversion in our organization, we would have to buy a subscription to some support, and we did that for the first year. We paid less than 1% of what I would’ve paid to upgrade my commercial software.

And guess what?

In one year, we never needed to call anybody, because nothing broke. We moved from commercial software to commercial SCM software and Subversion.

The other big thing was Maven.

We used Ant, and everybody had their libraries in their source control and the Ant script. It took days to even stand up a project in individual developers’ laptop. Building was also a nightmare. We decided to use Maven instead. We put Nexus and put all the libraries in there, and we’re going to resolve all the dependencies to our CI pipeline against Nexus — an in-house Maven central if you will.

But that posed another problem. How do you get the libraries? Before coming to Maven, this was our way to get libraries.

This an open source acceptance form. You fill this out — you print your name, your title, the date, and identify the open source software. Each and every library used to have one of these. They were multiple pages, I just printed the first page for demonstration purposes.

Then there were some pretty important questions asked from you. “Are you going to modify this software? Yes or no?” If you said ‘yes,’ you are in trouble. And then, “Do you intend to distribute this software along with your source code outside of Capital One?” etc. Then you would have to print that form, get it signed by your immediate manager, and your VP. Then…we faxed it.

Once the fax got there, the legal department looked at it. But the real question here is, why would we need to fax it? Back in the days, and maybe even now, our legal department was in a part of a building that is a little secluded. There’s a huge glass door, and you need a special badge to get in there. That’s the first reason. The second reason is, you could not reach out to the legal department directly. You couldn’t call them. Your email would not be answered, they’re busy.

Once I was walking past that building, I saw one of my colleagues standing in front of the glass doors and knocking, carrying a big bunch of printed sheets. I said, “What’s going on here? Why are you here?”

He said, “I have these open source library’s approval to be obtained before I can release, and my release is coming tomorrow.”

I said, “Why do you need their approval now?”

He said, “If I don’t get approval for this, the Change Approval Board won’t approve my release.” And he was standing there knocking on the door. He did finally get approval.

He and I decided that it doesn’t need to be this way, there has to be a better way. No engineer should go through this. We decided that we’ll set up meetings with our legal department to talk about the open source approval process.

We Learned a Few Very Important Things

First of all, they didn’t know why we were going to them to get approval for open source, and we didn’t know what the risks were. We realized we just had never talked to each other. We sat together and understood what they mean by ‘approval,’ they understood what we need open source software for.

Now, Jaime’s going to share about our open source intake process, and all the risks.

Jamie Specter

First, the goal of the DevOps Enterprise Summet is just to give the leaders the tools and practices that they need to develop and deploy software faster and to win in the marketplace. I want to provide you with that information so you can have those discussions about using open source.

What I’m not here to do is provide you with specific tools and processes to implement in your companies. You are the expert for that. You know what works best for you. Equipped with the right knowledge, you can take advantage of the value that open source brings.

At Capital One, we take great care in writing the code for our company. We do extensive due diligence in procuring our commercial software. Why wouldn’t we treat our open source in the same way? We want our applications to be healthy, and so we want to make sure that we’re protecting our companies no matter what we’re bringing into our environments.

I Bet You’re Using Open Source

First, we need to understand how much open source we’re using. I’d bet that you are probably using open source. Now I’m not usually a huge gambler unless I have some inside knowledge, and I do have a little bit of inside knowledge that I’ll share with you.

This data right here is based on a Black Duck survey of over a thousand code bases, in every industry you can think of.

What they found, is that 96% of those applications contained open source. Of those 96% applications that contained open source, 57% of the code base was actually open source.

This visual here will drive it home.

Yes, you can think about going out to a repository, downloading the open source, and bringing it in. Open source is coming into your environment from a number of different avenues.

The takeaway from these two slides is one, basically all applications contain open source, and more than likely, on average, your applications are containing more source than even your proprietary code. So it’s really important to understand the risk.

How Do We Think About Open Source?

Open source is free, meaning that there’s no initial monetary cost, and it’s free as in freedom. You can use, distribute, modify the source code without going to ask the author for permission. Something else you may have thought of is open source is free as in a puppy. Meaning, who doesn’t love puppies, they’re cute. They make great friends. They just make you smile.

But having that cute companion that makes you smile can actually come at a high cost. They require a lot of work. You have to feed them. You have to clean up after them. Many people, you’ll see, still have dogs despite the obligations because there are ways to make it work. There are ways to make the process manageable.

Open source is the same way. Understanding those risks and obligations is an important part to ensure that you’re able to be prepared, that the right people are involved, that the right tech is used, the right processes are in place.

Risks: Security

Now, I’m the lawyer, so let’s talk about the risk

The main thing I want you to know is just the fact that there are real risks using open source. Someone in a legal or risk function that’s not as familiar may have some hesitancy. It’s easy as always just to say ‘no’ versus taking the time to understand. But hopefully, after this, you’ll be ready to have that conversation because they’re equipped with the right knowledge.

There are two buckets of risks. First up, security.

Let’s look at the takeaways:

Number one, your developers are the first line of defense. They are the ones bringing in the software, so they should be the ones managing it. There needs to be that ‘you build it, you own it,’ mentality embedded into your culture, which is consistent with the DevOps mindset. Also, unlike commercial applications that push those updates for you, your developers are responsible for pulling those updates.

Since vulnerabilities can be found anytime, and we do see breaches continue to increase, that continuous detection and remediation is necessary. You do not want to be a part of that team that has to answer to your CEO and the executive team because there’s been a breach for having waited over 1,500 days to resolve one security vulnerability. Remediation and security roll just means to upgrade, usually, to the most recent version.

Security: License

Now there is also a real legal risk. Some things to keep in mind, you need a license if you’re using open source. That is your legal permission. Those are the rules. Authors of software actually have over 2,000 different licenses from which to choose.

When we think about open source licenses, we can think about it in two different buckets. We have permissive and copyleft.

When it comes to permissive, generally the obligations are to give attribution to that author, you want to give them credit.

The other end of the spectrum is a little bit scarier for organizations, and that’s the copyleft licenses. In certain situations, you not only have to distribute the source code and show source of the copyleft licensed software that you’re using, but you also have to share the source code of any code that it touches. It’s a very use-case specific analysis.

I also want to point out that you’re using the open source that you’re bringing in as-is. Meaning, that you have no recourse against the author from which you got it. That’s also a tricky area to explain to your legal and compliance functions because with commercial you do have those reps and warranties.

When we talk about remediation, you upgrade the version. It’s a pretty straightforward approach. With licenses, generally they aren’t going to change throughout the course of the project, so remediation in that sense is a little bit different. It usually involves replacing it with an open source alternative if you can’t use it, something you write yourself, or a commercial alternative. Also with remediation because the obligations are usually conditional, meaning, ‘if I use the open source this way, then X.’ Since it is conditional, you can work with your counsel to get approval if the use case actually mitigates the risk that those obligations will take place.

Sometimes, if you don’t know what the license is, instead of requiring removal, it’s easy to get on GitHub to request confirmation from the author. That’s all that you need. If you want to see if it’s a license that you can’t use internally for whatever reason based on your organization’s risk level, then you can just get online, and get on GitHub, and ask them, “Hey, let’s maybe add a permissive license to that.”

Then last but not least, we’re all busy. Sometimes you just have to submit a pull request with that license that you’d like them to adopt, so it’s just really easy for them to add.

Worst Case Scenarios

Speaking of attorneys, what’s the worst case scenario? Here are the kinds of things that you want to tell your leadership so that they can know why it’s urgent to get this stuff in place.

• Code secret disclosure. Your code is your secret sauce. If you have to open source your competitive advantage, what’s left for your company’s worth?
• Devaluation of patent portfolio. This is for companies that have patent portfolios. Some open source licenses will have patent grants, meaning that you’re giving out a free license in certain situations for the functionality. From a policy perspective, this makes perfect sense. If I’m throwing out some open source software for the community to use, and then everyone’s adopting it, it’s great. But, if I have a few patents in my back pocket, we want to avoid having them go sue everyone for patent infringement. That’s what the licenses do, but you should be aware of this if you do have a patent portfolio because the value of patents relate to your ability to exclude others from using it, so it will devalue the patent portfolio.
• Mergers and acquisitions. For the companies here looking to buy other companies or the companies here that are looking to hopefully get big enough one day to sell. It’s very important to be in compliance with your open source obligations. Open source compliance issues can delay a deal. They can devalue your company, and they can kill a deal altogether.
• Security threats and potential fines. I have Heartbleed and Equifax right there. You don’t want your company to be a part of a national breach and get that much media attention.
• Reputation risk. Your customers, your current employees, your potential hires, don’t want to work for a company that has a bad reputation for risk management. Especially when you’re in an industry like ours, with the financial services institutions.
• IP infringement. IP is intellectual property, not internet protocol. Typically litigation in the open source space has been from open source foundations, and they’re just really motivated by just making sure that people are compliant with their licenses. But we have seen an increase in the other side of the category of people that bring litigation, and those are motivated by businesses.
  
  Here is an order from a recent court case that recognized the copyleft license as both a license and a contract.

And what is important to keep in mind about this is that it opens up additional remedies that a plaintiff can bring against the defendant that they’re suing.

Something else to think about here, what happens if because of pending litigation you cannot use your mobile application, you cannot use your browser extension, and that’s your main offering? Or what happens if you’re forced to comply with the copyleft licenses, versus substituting it out, and you have to open source the proprietary code? Both of these scenarios can have a big impact.

Best Practices

Now we understand who’s using open source (everyone), and we understand some of the high-level risks, so how do we think about tackling those risks?

This is a framework to think about as you kind of start the journey, depending on where you are. Making sure that you know what you’re using, then assessing it. There are some solutions out there that you can map different security vulnerabilities to, for example. But just think about it as ‘identify, analyze, and remediate.’ As always, there is going to be that continuous monitoring and auditing, which can fit very nicely into your DevOps pipeline.

Mitigate with a Governance Strategy

Capital One has implemented every one of these.

I spent a lot of my time on the last one, education and training. I’ve also included two links at the bottom that are helpful to reference no matter where you are on your open source journey.

Now, Topo’s going to help us understand how DevOps can help.

Topo Pal

Open source helps DevOps, DevOps helps open source, too.

• It allows you to automate. Now, do not automate a bad process, because if you automate a bad process, then you get bad automated processes. Instead, look at your process and try to determine which one is more important to you, or which one makes sense, and automate those. Shift left, right from the beginning of your pipeline, start looking for open source license violations or legal violations, and mitigate that accordingly.
• Frequent releases. If we need to patch something, we can always frequently release that patch as soon as possible, instead of waiting for six months.
• Smaller batch size reduces the risk. It can only release that particular library object version to your production.
• Collaboration and transparency is a critical thing. Without this, you cannot go farther.
• You cannot do it alone. You need to bring in engineers, legal security, risk officers, and anyone else you may need along your way. Without collaboration, there’s no way you can get there.

Lastly, Just Using Open Source Is Not Enough

Remember I talked about our CI journey, how we put all the libraries into our Maven depositories? While doing that, we found an incredible thing. There are libraries that look like open source, but they’re not really open source, because I modified them, and on that form, I said I wasn’t going to modify them, I lied.

Those modified libraries are in those source code repositories. So what do you do with those? You cannot contribute it back, but we cannot use it. I knew there had to be a way.

I started proposing this idea that we should contribute back to open source. And that was pretty disruptive. I thought that deception is bad. So what did I do? I posted in our social network within Capital One. An article blog post, “Should Capital One contribute to the Open Source community?”

At that point, we had a hothouse. The whole idea of hothouse is that all the decision makers make us come together, and try to resolve the issue that deliverables are not allowed to contribute to open source. In that hothouse, we decided that, yes, we are going to make contributions and that was a good day for Capital One engineers.

That was four years ago since then we have contributed to open source 90 developers, 193 projects. Most notable are Angular Ansible and Hadoop Block for Jspark and etc.

And we didn’t stop there. We have our own open source projects, for three years, a total of 31 projects. Most popular are Hygeia and Cloud Custodian. Many of you already use these tools and contribute back.

This is our open source page.

We are also working at many enterprises, to do what you call entersource, the next version of open source. What that means is enterprises getting together and solving their own problems without having to rely upon commercial vendors. We have partnered with companies like Verizon, Walmart, American Airlines, and forming a community around the DevOps practices.

Jamie Specter

For us, being open source first results in a lot of benefits, but the one I want to call out for you first is it created a culture of collaboration.

Perhaps the most valuable observation I’ve made while being at Capital One and also working with Topo is how much culture can influence behavior to do the right thing. The right thing for you, the right thing for your company, the right thing for your customers. And it’s not an easy journey. There are a lot of questions. There are multiple risks to manage. There are a lot of lessons learned. But with the right knowledge, the right team, the right technology, the right processes, it’s been an incredibly valuable journey so far.

Topo Pal

Here’s my final appeal: Make it easy, streamline, automate as much as possible (not the bad processes, but the good processes,) collaborate often, and create a good engineering experience.

This was an excerpt from a presentation by Capital One’s Topo Pal, Senior Director & Sr. Engineering Fellow, and Jamie Specter, Counsel, Capital One’s IP & Technology Legal Group.

You can watch the video of the presentation, which was originally delivered at the 2018 DevOps Enterprise Summit in Las Vegas.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
•