Skip to content

September 22, 2018

DevSecOps (Expert Panel Discussion)

By IT Revolution

The following is a transcript of an interview by Alan Shimel of with three leaders in the DevOps community — Margo Cronin, Zane Lackey, and Ilkka Turunen. This interview was conducted during the DevOps Enterprise Summit in London 2018. The topic at hand is DevSecOps.

Some of the responses have been edited for brevity and clarity. Watch the full interview here.

Alan Shimel: Great. So you know, we are going to explore a little into DevSecOps, and the role it’s playing in today’s DevOps stack if you will, right, because one of the things we’ve explored over the last two days is those things we spoke about in 2014, 2015, and 2016 about DevOps, it’s not that they’re any less true or not true anymore, but they’ve evolved, and we’ve learned some lessons. In classic DevOps, we’ve taken that feedback loop and maybe changed a little bit.

But Margo, I thought we’d start off with, you spoke here today, what did you speak on?

Margo Cronin: Well, one of the points I touched on was that now digital regulation is getting much tighter, DevOps has evolved so that it has to include security. So, I began off by saying, personally, I’m not crazy about the term, I’ve never been crazy about the term DevSecOps, because it’s like Sec is an afterthought. You know we were squeezing it in between Dev and Ops. The last kid that got on the bus, and we’re like go ahead and just sit in there. And that’s not the case, right?

For all of us, security is the first priority. Like the top job, ‘job zero’ we sometimes call it. And therefore, DevOps understands that security is key and it is the first thing that you do. And if you look at GDPR, one of the pillars of GDPR, is privacy by design, so not just data portability, data breach notification, the right to be forgotten, but privacy by design. And it kind of mandates that you have to do DevSecOps.

Alan Shimel: Yup. So first of all, let’s do this, who likes the term DevSecOps?
Margo Cronin: Not me.
Zane Lackey I think everyone tolerates it.
Ilkka Turunen: Yeah, I think it’s like the best of the least bad bunch.
Alan Shimel: Now, we’ve all talked before — Margo, and I started in security in the .com days, the late 90s, founder of a security company in 2001, and basically have been doing what you’d call cybersecurity or security ever since. And I got into DevOps only because of security. I thought this was going to be a great thing for security.

I ran the security bloggers network for 15 years, and I’ve done a lot in the security community over my time. And,

I’ve gotten into huge fights, huge fights with my friends in the security world over this DevSecOps thing.
Some people just say it’s flat-out nonsense. Alright, it’s a marketing term.

Other people and it just goes to show you how weird we are in security. Other people say “DevSecOps, why isn’t it SecDevOps? We are most important after all.”

Other people, you know, want nothing to do with it. Or, they’ll say SecOps, we don’t really work with the Devs, they were assholes, they were always that way. You know,

I don’t want to work with them. And that again is part of the problem as well. It’s a tremendous controversial thing that I personally had to get my head wrapped around. Did you have a journey here, Zane?

Zane Lackey: Yeah, I mean, the naming part aside — I think the broader movement of security being a piece of DevOps is critical.

Right? If the way that we are creating software has changed, and the way that we’re delivering software has changed, well, the way that we’re actually securing that and defending that has to change as well. And so, whatever we want to call that, I could care less.

Alan Shimel: Ilkka what’re your thoughts on that one?
Ilkka Turunen: Well, I mean, I’ve actually come to it from the other end of the table, like form the development floor, and then from the operations floor. To me, one of the big problems always was security, and it was kind of like a point activity.

It was seen as a thing that you have to do to clear the boxes. I think the main aspect of this whole conversation is are we going to change that logic and think about it more almost as an engineering discipline, or as a way of just doing our work better?

It kind of ties in nicely with all of this engineering craftsmanship and software craftsmanship movements, as well. To me, DevSecOps is the best way of conveying that message, it’s like it’s a term that everybody kind of understands, no one can agree upon it, which was what happened to DevOps, but at the same time, it means that people have an opinion about it and they can communicate on it.

Alan Shimel: Fair enough. I mean, I know in my own personal journey, I put on a DevSecOps day at RSA Conference in San Francisco every year.

We’ve done it for the last four years, and we called it ‘the Rugged DevOps day,’ then we called it just DevOps and Security, and then the last three years, DevSecOps.  
So, let me tie a bow on the issue. I’ve come to the conclusion, like you Zane, that I don’t care what you call it.

DevOps is DevOps. There is only one DevOps. There is only one truth. I don’t care what you call your God, there’s only one truth, right? And it’s DevOps. Now, if we need to add security to the term and stick it in the middle as you say. Because that’s going to satisfy people and make them think about security, so be it. I’m not going to stand in the way of that, I think that’s a pragmatic approach to it.

Zane Lackey: Sure. Well, I think the journey of security into DevOps makes for a very long and bad hashtag. got to call it something.
Ilkka Turunen: Again, I think DevSecOps just fits in the mouth, doesn’t it?
Alan Shimel: Yeah. It does and it seems to roll.
As a result of all of this, I’ve gotten involved with the Sonatype folks, Mark Miller, we’re doing DevSecOps days with John Willis and do it at RSA, we’re going to do one at RSA APJ in Singapore. We’re going to do some in Australia, and Bangkok, and hopefully more. We’re doing one in London too. So we’re hopefully going to see more of that type of thing. But let’s throw the name out for a second and let’s really focus on what it is, and I think Margo, you touched on it right in the beginning.
Margo Cronin: Yeah, well I mean, one of the things I was wondering about is how do you get everybody within the team to become a security practitioner? I mean, you were kind of touching on it there, Ilkka, now everybody’s a security owner as well. Interestingly, everybody has become the way you (Alan) were in the late 90’s. Everybody is that cyber security guy. And how do we get that mentality into the DevOps team? Into the scrub room?

One of the things I was talking about in my speech is, it doesn’t matter if you’re doing site reliability engineering, or if you’re doing DevOps, etc. let’s just throw everything to the wind, and security has to be first. So how does everybody become a security practitioner?

Zane Lackey: Yeah, and I think to that point, the existential shift that is happening for security right now is that it’s going from exactly what you said, this outsourced group that was a point along the journey, to a fundamental role in an organization. This changes the question to, how do we make make the rest of the organization’s security self sufficient? And that is a fundamental change in the way we have built and run security programs for the last 20 or 30 years.
Margo Cronin: Yeah, it’s not a chore.
Ilkka Turunen: And that mirrors very much the transition that Q & A went through. So if you think about testing ten years ago, it was literally people running through test execution plans, and gradually they changed from that to becoming writers of tests. The very people that write the automatic execution and help the floor become more efficient at testing for themselves — teaching them unit testing, all of these other frameworks. So I feel we are at the brink of a similar kind of change. I think it’s a mixture of both in sense of psychology and just changing roles, like, no longer do we want to be adversarial with each other.
Alan Shimel: Yeah, we definitely don’t want to be adversarial, that whole adversarial thing is so antithical to what DevOps is. We’re not adversaries here.

The other thing I would mention, it’s funny you mentioned Q.A. because I’ve had this conversation with someone else in the security space, but maybe what the goal here should be is that security become synonymous with quality. Alright, instead of thinking ‘oh, I’ve got to add the security in,’ we can’t have quality product without security.

Zane Lackey: Yeah. The best way I’ve ever seen like the highest performing organizations view it as security is a subset of good engineering. In the way that reiliancy is, reliability is, quality it, performance is. It is a subset of good engineering.
Margo Cronin: But not to argue that point, but maybe to give it another perspective, certainly the industries I work with are highly regulated and highly audited. So it’s not so much about quality, of course quality is incredibly important. But I know when I’m sitting with development teams and with ops what they’re concerned about is regulators. And where I come from, Switzerland, and in banking, they’re not really concerned about quality much at all. For them, I think it’s a given, you know, it’s Switzerland, we’re going to do quality.
Alan Shimel: And this to me has always been the problem with compliance. I call it least common denominator security, right?  Because, frankly in so many times that the compliance regulator people are the low bar, it’s not what we should be aiming at, but we need to check the box.
Margo Cronin: And how do you move it from a box-checking activity, a “chore”, to kind of actually something everybody owns?
Zane Lackey: This is where I completely agree with your point there. I would actually see it like this, security and compliance as two circles on the venn diagram, and there’s certainly some that overlap and some that are completely separate there.
Ilkka Turunen: And I recognize that exact same movement over the last couple of years. I’ve had a pretty good insight into this kind of larger vendors, in heavily regulated industries. One of the things you actually notice, we publish a report every year called the State of DevSecOps ourselves. The last one was about how there’s a hundred developers, to ten operators, to one security person ratio in a standard enterprise. If you just think about it from a number perspective, what can one person do against a hundred?

So it very naturally drives them into that, we got to do the lowest common denominator because that’s literally the only control point that we have.

Alan Shimel: Yeah. But it drives something else out there, because it’s not that one guy’s mission to do all the security. It’s his mission to turn on the ten and the hundred so that we’re all responsible for security and we’re all thinking about what that regulator says or needs to see.
Margo Cronin: Or it’s his mission to drive automation. His mission that everybody thinks about automation. I did touch this a bit in my speech, when you’re using tools and vendors, consider the effort for security automation as part of the work that you’re doing. So your product backlog, or however you describe your story should reflect the security automation you need to do as well.
Alan Shimel: I don’t think anyone’s going to disagree with that, right?
Ilkka Turunen: And from a development point of view, I almost see as analogies as testing, that’s in reality what we should be seeing the kind of security activity as. If you think about what the fundamental shift in software engineering was, it was like ‘let’s all do unit testing, all do other sorts of testing across the line,’ and that took a long time, people resisted that change.

This is almost like a very similar transformation. It’s layering on that practice, to say we consistently need to execute on this, we consistently need to have this along the line. Then, obviously, still have all the checks we have today, but more as a kind of verification point of the work that you’ve already done, rather than relying that’s going to be the only thing that we’re going to do.

Alan Shimel: I don’t necessarily disagree with you, but I think one of the lessons we’ve learned at DOES this year is that sort of the next frontier in DevOps is moving beyond that continued shift left, to not just the testing phase but, let’s move security, let’s move DevOps and security along with it to to the planning stage of the business. Because if we don’t think security when we’re planning the thing, we’re still in that bolt-on mind frame. Security has to be organic and if it’s truly going to be organic, it has to be done at the planning stage. Agree?
Margo Cronin: Agreed, but do you mean at the business planning stage? I’m thinking aloud here, but would they really have an understanding, you know if you think about something like D-DOS attacks, or keys going into GitHub, as a business owner…
Alan Shimel: You’re right. But maybe it’s the role of the CSO or maybe it’s the role of that one poor S.O.B. on your team, who has to translate the D-DOS attack into what that means to the business owner, which means your app is going down. But you know what I mean, we need to bullet proof it. So, and this has always been a problem in security, Zane, I know you have dealt with this, right?
Zane Lackey: Sure.
Alan Shimel: Which is, how do we translate the security jargon…
Zane Lackey: Into business speak?
Because I completely agree with your point, there needs to be the discussion of business risk, but I think this is something that I see go wrong in so many security organizations — they don’t make that jump from technical risk up to actual business risk. And so they’re saying “oh, well, great, you’re building a new thing, well, we’ve got to be worried about phishing”, like okay great, what does that mean to the business?We’ve got to be worried about the loss of our customer information here, or the service not being available, let’s discuss that. Let’s put it in the context of you know of, perhaps a breach or an incident that the Executive leadership has heard about, and map that to our business.That way we can say, ‘we’re going to do this new application that’s going to be a new way to interact with our customers. The risks of that are similar to this particular breach that we had seen in the past, so these are the mitigations and the controls we’re going to put in place for that. Here from the compliance side, these are the things we need to be worried about operating in these different environments,’ and not just saying, “oh, we need to buy a firewall, and we need AV.”
Alan Shimel: …Or we need a D-Dos solution. That doesn’t work.
Margo Cronin: Yeah, because I think if traditionally you kind of think of it more like electricity. It should just work. And maybe when it comes to say business risks like data, that kind of thing, there is some understanding there, but trying to translate that can actually be I think sometimes quite tough.
Ilkka Turunen: There was a talk by George Corman, many years ago, I remember in the rocket days that kind of spoke about incentives in business in general, which is if you think about typical software deployable, there are like three main incentive. Deliver on time. Deliver on budget. And then deliver with acceptable quality and risk. And that kind of tells you how much of an afterthought the whole thing to the business is. So I kind of completely agree with you.
Alan Shimel: But is that still true today?
Ilkka Turunen: I think it is. I mean it is definitely getting better as we get more aware of it, but it comes back to the point rather than, there definitely needs to be a dollar sign on the risk and you do need to bring it to the board. I almost see it as a wrap up of quality itself. In the same definition of quality that we have in cars, we should have in software, and a part of that is exactly this, and then frame it back, ‘well, okay this accident happened, we have this call back, it cost this much.’ Can kind of start putting dollar signs on the whole thing?
Alan Shimel: Fair enough. I mean, so let me go in another path for a second if I can.
So we’re talking in essence about shifting left. Moving it all the way to the business phase, to the developer, and all of that. But we also have an issue, and we think about things like Equifax, and some of the other high-profile breaches we’ve seen this year. Where, we do the app, it’s “secure” we release the app, and then lo and behold, we use struts, and we just found out there’s this huge hole in struts, or there’s some other open source component, because let’s face it, most of our apps today are 75-80% components with you know, 20-25% custom code gluing it together. Is that part of the mission? It should be in my opinion.
Zane Lackey: Absolutely.
Alan Shimel: What do we do? How do we secure these?
Zane Lackey: Yeah. I think, to potentially coin a term here, I think at the same time we are shifting left, we need to be shifting right as well, which is part of what made the whole DevOps movement so successful. We need to focus on visibility and feedback loops and the ability to iterate on top of things. And, the way you do that is by having awareness of what’s actually happening in our environment. For a lot of security, that awareness just isn’t there. Right?
Ilkka Turunen: Those are my views as well. I work quite a lot with helping manage in this supply chain. That’s exactly what it is. And you know, it is a software supply chain, you have producers of software that you consume into your application, open source or closed source, and those people have producers, and then those people have producers.

Part of the problem there is that particular vulnerability was in one of the rest API plugins that had a transited dependency. So, it wasn’t immediately clear, even if you looked at the direct manifest, that that was even there. So the challenge of managing this supply chain is, the only real visible solution is to essentially look at the actual deployables, print out everything inside of there, maintain those lists, and then correlate on them.

That’s one of the big challenges we have when addressing those sorts of issues. When we look at the Equifax case was it took them over, it took them however long to actually patch that out, I don’t think they ever did. But it only took them about two or three days to get owned by the vulnerability once it came out and the project released a new version of it. So, I remember from my experience and the typical way that worked, was we found something bad that had happened and then we had to have a fire drill to find out where it was.

There’s much better ways to start automating on that, creating what we call ‘bills of materials.’ And then just building a database. So when something does go wrong, an event happens, you can still use that as searching function and find out. It’s these small sorts of things that make sense.

Alan Shimel: Well, let me propose something radical, and I’m going to get your opinion on it Margo. Is there a role there for the public cloud provider to help us kind of uplift the security of this?
Margo Cronin: Absolutely. And I think in every aspect from you know, being part of the DevSecOps team, and you know helping everybody, become a security practitioner. And these concepts of automating, then moving into this.

Again it’s kind of touching this CO, are we moving to the let conversation about is it okay to fail, and then recover quickly? First of all we need to prevent, but if prevention hasn’t worked, and the failure, the incident has occurred and reducing glass radius so you can recover quickly. And then as a final step, which I think will be really interesting in the future is machine learning. And how we can use machine learning to actually change the behavior of our applications and production, and so they can start training themselves.

Alan Shimel: You mean the machine learning is not here yet?  What are all those people selling me?
Margo Cronin: Yeah, it’s in its infancy without a doubt. I mean, if you look at services right there in machine learning, you now have services that can scan your landscape and say do you know these log files contain client identifying data? Do you know you have keys there?

Then you have services where you can then change that behavior automatically. But I think that’s where we’re going to see cloud service providers become a lot more active, you know, using machine learning to harden your production landscape before you actually even go to the security operators.

For me, it’s kind of like Ilkka, you were talking about there like, are we comfortable with things going wrong in our production landscape and recovering quickly? You know, Richard Cook gave a great session yesterday evening on where he talked about, he didn’t say meantime to recover was nonsense, but he was quite, he was quite critical about it.

Well, he was critical about a lot of stuff. But he, you know, he pretty much said, ‘don’t talk to me about meantime to recover, meantime to restore, because you don’t know what’s happening under the hood.’ I think is essentially his message. So, is that something we’re comfortable with, like things actually going wrong and recovering quickly?

Alan Shimel: Yeah, so I think, you know, that is a transition I have seen in my time in security which was from putting 100% of your resources into preventing an incident, to allocating some percentage of resources towards ‘event management’ if you will, or remediation, when you know that incident takes place.
Zane Lackey: I think ultimately, you have to be comfortable with it because it’s going to happen whether you’re comfortable or not. Right?

This was such a journey for me as a CSO of, being one of the first go through the whole DevOps and cloud shift is recognizing that the embarrassing lesson I had to learn was that look. No matter what our software development methodology or delivery methodology, we’re always going to have bugs, and we’re always going to have outages and issues, and all that. The system that allows us to react the quickest, and gives us inherently the most visibility is the one that actually makes us safest.

Ilkka Turunen: Again, I think, tying this back to the previous questions while in the supply chain type of situation, the biggest killer isn’t the new, bad events that happens today. It’s forgetting about the thing that happened a year ago, you know, continuing to use an old version of struts or spring or whatever that has a known violation, that you haven’t patched because it still runs. While it still provides the business function, it just has a new hole. So it is actually, I think like the healthy balance, there is a healthy diet here, like a food pyramid of healthy security you know that allocates between like disaster recovery. That’s why we have fire drills, that why we have to have incident response as well.
Alan Shimel: This has been a fascinating discussion guys. Thank you for your time.

- About The Authors
Avatar photo

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

No comments found

Leave a Comment

Your email address will not be published.

Jump to Section

    More Like This

    The Frictionless Dev Experience
    By David Anderson , Mark McCann , Michael O’Reilly

    This post is excerpted from The Value Flywheel Effect: Power the Future and Accelerate…

    Sustainability in Software
    By David Anderson , Mark McCann , Michael O’Reilly

    This post is excerpted from The Value Flywheel Effect: Power the Future and Accelerate…

    Serverless Espresso: A Case Study of Serverless Event-Driven Architecture
    By David Anderson , Mark McCann , Michael O’Reilly

    What is Serverless Espresso? Serverless Espresso is a pop-up coffee shop that allows you…

    Beyond Agile Auditing: An Introduction
    By Clarissa Lucas

    This post has been adapted from the Introduction to Beyond Agile Auditing: Three Practices…