Shifting Security Left at Fannie Mae

This case study has been excerpted from the second edition of The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois, John Willis, and Nicole Forsgren, PhD.

---

Fannie Mae has a more than $3 billion balance sheet and helps finance approximately one in four homes in the US as of 2020. At Fannie Mae, safety and soundness is part of their mission.

They’ve experienced crises before. With a low risk tolerance, their challenge was ensuring that security strengthened everything they did. DevOps provided a solution to learn from chaos engineering to improve security, put security in the pipeline, and weave security transparently into the fabric of everything they did.

Chris Porter, Fannie Mae’s CISO, and Kimberly Johnson, the Executive Vice President and COO, talked about their evolution at the 2020 DevOps Enterprise Summit. It boiled down to two key changes: changing culture and changing the way security communicated with Dev teams and how they integrated security tools.

In the old way, Dev would hand off code that was ready for production. Security would conduct their own tests and send back a list of vulnerabilities that the Dev team would have to correct. It was inefficient and no one liked it. They needed to learn to shift security left.

They did this by relinquishing control over their security tools, making them more self-service,  making them API-based, and integrating them with Jira and Jenkins. They trained developers to run the tools and to learn what the results meant, and they had to change their own nomenclature (instead of vulnerabilities, they talked about defects).

They also had to fully integrate all security tests within the CI/CD pipeline so that every time code was checked in they were running a test. Ultimately, this made it easier for developers to know what to do. They could see a test fail, understand why, and fix the problem.

“I call this the paved road. If you follow the paved road and you use the CI/CD pipeline, which has all the checks integrated into the pipeline, then it will be easier for you to deploy code,” says Chris Porter.

This was treated like an Andon cord. If the test didn’t pass, then it broke the line and had to be fixed before the line could continue. If you didn’t use the paved road, it was a much slower, bumpier journey.

According to Porter, a mindset change is needed from development and security. In the past, security’s mindset had been to protect developers from themselves. But in a DevOps model, the work has moved to “you build it, you own it.” Everyone has shared accountability, and security is baked into the code instead of being inserted later.

As Kimberly Johnson put it:

In the old way, with Dev handing off production-ready code to Security for testing, we had a major bottleneck in the throughput of the Security team. For large organizations that operate at scale, it can be really hard to find enough Security talent to continually test everything that is developed. Building the security tests into the development pipeline unlocked a lot more productivity for us and reduced our dependence on Security personnel for standard testing and routine deployments.

In addition to reducing our reliance on the Information Security team, shifting left and automating our testing has yielded better business results. Our deployment frequency has increased by 25% in the last year, and our deployment failure rate has fallen by about the same amount. We are getting critical business changes into production much faster, with fewer errors, using fewer resources, and generating less rework. Moving to DevSecOps has been a win-win-win for us.