Here’s How The Amazing Twitter Infosec Team Helps DevOps

Want to see how infosec integrates into a DevOps work stream? Watch this fantastic talk by Justin Collins (@presidentbeef), Neil Matatall (@ndm), and Alex Smolen (@alsmola) from Twitter, called “Put Your Robots To Work: Security Automation at Twitter.” (Slides are here and video is below.)

Watching this talk should be required for any infosec person who wants to see how they can integrate into the daily work of Development and IT Operations.

Of course, they mention what it’s like to be at an organization going through hyper-growth (i.e., the famous “Fail Whale” due to capacity issues). But much to the amusement of all of us, they describe the birth of the Twitter infosec program, triggered by the hacking of the @barackobama account.

That breach resulted in an FTC injunction, requiring Twitter to be secure for the next 15 years. And hence, Twitter infosec was born.

This newly-formed department made some huge strides during the Twitter Hack Week, which occurs once every quarter, where they were able to focus on proactive work. During Hack Weeks, everyone “works on whatever they want, which they then demo to the rest of the company,” similar to what’s done at Facebook. These projects often focus on work that reduces technical debt, automate manual work, improve processes, etc.

They wanted to focus on creating more automation, but anchored in the these framing principles:

• Get the right information to the right people: “writing secure code isn’t a technology issue; it’s about people working with technology.  So how do we communicate better?  We didn’t want to do something that generates a huge report, and then just send it to someone.”“Instead, we need to provide the developer who created the vulnerability exactly the information they need to fix it.”
• Find and fix bugs as quickly as possible
• Don’t repeat your mistakes: “the best predictor of the next bug is the last bug. We found that we were fixing the same bugs over and over. Our goal was to use automation to prevent them from happening again.”
• Analyze from many angles: “we gather data from the source code, from the production environment, and even the user browsers.”
• Let people prove you wrong: “we need Dev to trust us, so they need to be able to tell us about false positives. If we’re wrong, we want them to let us know. And we don’t want to waste their time.”
• Help people help themselves: “most people want to do the right thing, so give them all the context and information they need to fix it and they will. “
• Automate dumb work. “we define ‘dumb work’ as anything that doesn’t require judgement or creativity.”
• Keep it tailored: “building our own tools has allowed us to save time and money, because it allows us to look for the narrow things that we care about.”

Automating Security

Justin Collins spoke about the manual security tasks of reviewing code, penetration testing and handling reports from the external world. They set out to automate all of these activities. He made a fantastic point of the workflow around static code analysis:

Even though the static code analysis step is “automated,” infosec still has to do a lot of waiting: waiting for the scan to complete, get back the big stack of reports, interpret the reports, and then find the person responsible for fixing it. And when the code changes, we have to do it all over again! Even though we’re using ‘automated tools,’ we’re still doing a lot of manual work…. So we wanted to put our robots to work. By doing this, we do less dumb button-pushing tasks, doing more stuff with creativity and judgement.”

Back to that first Hack Week: they built static code analysis into the Jenkins continuous integration process, but there was much more they wanted to do. So they set out to build SADB, the Security Automation Dashboard. The logo is, of course, a sad bee.

SADB takes input from brakeman, phantom gang, csp, threatdeck, roshambo, and the outputs include emails that go to developers and infosec.

Brakeman for static code analysis

Brakeman does static code analysis for Ruby on Rails. The primary author is Justin Collins (@presidentbeef ). It’s had 25 releases in last year. Brakeman runs at any of the following times, but in the ideal, Brakeman runs each time the developer saves code!

This is great. In typical Twitter development, Brakeman runs upon each code commit (git push), and when vulnerabilities are found, SADB will email the developer about how to fix it. Better yet, when the fix is committed, SADB will email them another email, congratulating them for fixing it!

Nice. Having infosec integrated into the development and continuous integration cycle is awesome. Catching it at “file save” time is just awesome.

What has been the results of using Brakeman? By creating fast feedback for developers, showing them how to fix their own issues, clearly they’ve made a difference. (The spikes are usually associated with new releases of Brakeman.)

In the video at 19:06, you can find a demo they gave of Brakeman. At the end of the demo, listen to the applause when the developer fixes the vuln, because they’re not “angry and confused” by the results of static code analysis run. Nice. 🙂

Note that the “false positive” button in the vuln notification page is labeled, “Bullshit!” Haha.

By doing this, Twitter infosec is enforcing that code is tested early and often. They aspire that Brakeman be extended in the future to deal with

Phantom Gang for dynamic analysis

Neil Matatall then described Phantom Gang, which does dynamic application security testing (DAST). It complements Brakeman by looking for issues like mixed content, sensitive forms posting over non-HTTPS, old versions of jquery that often pop up when new microsites are created, forms without authenticity token which are prone to forgery, etc.

Phantom Gang is a bunch of node.js processes which emulates Webkit browser sessions, which are spun up as headless browser instances to see what users see. The output of Phantom Gang goes to JIRA, as opposed to directly to developers directly. Why? Often the issues found by Phantom Gang are more difficult to trace to an individual developer.

Interestingly, Phantom Gang was inspired a conversation they had with the Etsy crowd (holy crap, they’re everywhere). They eventually want to extend Phantom Gang so that they can see the effects of any SQL Injection attacks.

Phantom Gang will eventually open-sourced.

CSP

“Twitter is a big fan of CSP. It’s great for enforcing policy and protecting web sites.” Here’s an article on how Twitter uses CSP.

In a typical XSS attack, the attacker injects arbitrary Javascript into a page, which is then executed by an end-user. When a website enables CSP, the browser ignores inline Javascript and only loads external assets from a set of whitelisted sites. Enabling CSP on our site was simply a matter of including the policy in the returned headers under the CSP defined key, ‘X-Content-Security-Policy’.

They’ve configured CSP to disallow Javascript on the page itself, and have Javascript only served from themselves. When there’s a violation of that policy, “it’s almost assuredly a sign that there’s a valid XSS attack underway.” They also use their Big Data capability to look for site spikes, which is often an indicator of XSS attack.

They also use CSP to enforce HTTP Strict Transport Security, which protects against firesheep attacks, which our Dev and Ops people love.

Other great gems from their talk

They also configure TweetDeck configured in a way to monitor the tweet stream, look for specific attack patterns.

Long ago, they played “Ro-Sham-Bo” (paper, scissors, rock) to pick who among the team would do the awful work of importing all the data into SADB. They now have automated all that painful work, but still play ro-sham-bo to assign who reviews all the SADB reports.

They characterized their journey as the following:

• from manual -> automated
• from low visibility -> trending/reports
• from late discovery of issues -> auto notification

• Question: “How long did it take you to get to where you are?”
• Answer: “We went from constant emergency mode to operating more strategically in a couple of years”

Again, watching this talk should be required for any infosec person who wants to see how they can integrate into the daily work of Development and IT Operations. Watch it now.

Video of talk here
Slideshare

- About The Authors

Gene Kim

Gene Kim is a best-selling author whose books have sold over 1 million copies. He authored the widely acclaimed book "The Unicorn Project," which became a Wall Street Journal bestseller. Additionally, he co-authored several other influential works, including "The Phoenix Project," "The DevOps Handbook," and the award-winning "Accelerate," which received the prestigious Shingo Publication Award. His latest book, “Wiring the Winning Organization,” co-authored with Dr. Steven Spear, was released in November 2023.

Follow Gene on Social Media

• 
•