Interview with David Ashman, Chief Architect at Blackboard Learn

IT Revolution interviewed David Ashman on his definition of DevOps, his DevOps journey, and how DevOps has affected Blackboard. This is the first of a series of interviews with DevOps practitioners. For audio, click here.

Todd:  Good morning! Welcome to the IT Revolution Podcast. I’m your host today, Todd Sattersten, and today I’m talking with David Ashman. He’s the Chief Architect at Blackboard, and we’re going to talk about DevOps. Good morning David!

David:  Good morning!

Todd:  Let’s start with the question, this nebulous question that everybody has, about DevOps and what DevOps is. If I had to try to tackle you down and make you give me a definition of what DevOps is, what is that to you?

David:  It’s a great question. It’s a question that comes up all the time, and I feel like the one misrepresentation I hear the most—I’ll start with that, and then talk about what I see as the real definition in my eyes.

The biggest misrepresentation I always see is that DevOps is a team, and I see that used a lot, and it’s happened a lot at Blackboard where there’s a team labeled “DevOps.” It’s not really a group people. It’s not one person. It’s not a series of people doing certain specific things.

DevOps is really a culture. It’s a view and a perspective on how to do things more efficiently as a development in an operations organization. Historically, particularly at Blackboard, there has been a very strong separation between operations and development. Development did their work, we wrote our code, we wrote our tests, we verified our stuff, and then we packaged it up and we threw it over a wall to our operations team and we said, “Run it for us.”

That can work to a certain extent, but what ends up happening is you have this air gap between your development and operations organization that results in a lot of loss of fidelity, a lot of missed opportunities for things to get better for everybody.

And so, in my eyes, DevOps is all about reducing that air gap, getting it as close as possible, if not the same people, so that the people that are doing the operations have an intimate understanding of what the development organization has had to do to get to the point of having a product, and that the development organization has an intimate understanding of what the operations team has to do to make sure that application runs well, so the two organizations can work together to make sure that everybody benefits both internally and externally.

Todd:  You sit in an interesting spot being an architect there at Blackboard where you can sort of see all of these pieces and how they work together and have some influence over that. I guess my question there would be, what’s your advice to someone who, whether I’m trying to start from one side or the other—I’m a dev person, I’m trying to get something started, maybe I’ve been doing agile, or if I’m an ops person, I’m trying to eliminate some of that air gap. You sit in the middle of all of that, and it seems that there’s an interesting perspective there of you being able to talk about how to bridge that or close that air gap.

David:  I’m a very pragmatic architect and I feel like, no matter how many rooms you sit in with somebody upfront pontificating about how to do things or showing pretty graphs, getting things done is really the best way to learn how to do something. I’ve always been the kind of person that doesn’t necessarily read the book but just writes something. I’ll make a whole bunch of mistakes up front, but after a few iterations, I’ll get it right and I’ll sort of understand what’s going on.

I’ve always felt like getting your hands dirty is really the best way. Even if it’s just on the surface getting dirty. Taking it down, and say they have a question, “Well, I need to be able to deploy X, Y, and Z to see if it works okay.” All right, well, try it out. Here’s a server. Here’s the access credentials. Go ahead and it doesn’t matter if you mess it up. Go ahead and try it out and see what happens. “Oh, I can’t figure this out. Can you help out?” Then the two people working together figure it out. Giving the devs an opportunity to try and operate things and giving the ops people an opportunity to try and write some of the code that’s going into the platform.

To me, that’s the best opportunity for people to see both sides, because I think a lot of times people see it as more of the devs don’t understand ops, but I do think it’s both sides. I think sometimes the operators don’t always understand what really goes into making the sausage, what goes into really making that product that’s wrapped up and handed over.

So, on both sides of the fence, making sure that people can understand what everybody is doing. Even if it’s at a small grain of just feeling the pain just for a little while, gives them a little bit more sense and a little more interest in wanting to understand more about it.

I’ve been doing software for a long time and, even at Blackboard up until very recently, that has been mostly a development role. Here are a bunch of requirements, how do we make the software work really well? It hasn’t been until maybe the last year, year and a half, that I’ve gotten much more involved in the operations side of things, and the amount I’ve learned in the last year and a half of everything operations from hardware to software, to configuration management, to tooling, to monitoring—it’s incredible how much you can learn, even after spending fifteen, twenty years working in development. You don’t have any clue what’s going on on the other side if you’ve spent all your life in development. Just getting a glimpse for a really good engineer or really good operations engineer that has the interest, has that opportunity get that taste and then want to learn more.

Todd:  What’s the short story of your evolution in terms of seeing the gospel of DevOps, both personally and as well as what it’s meant for Blackboard and the company?

David:  DevOps really started to take off when we had a new team member join into our PD operations team. We had actually gotten to the point where we had fractured our operations into a PD, a product development operations team, that focused on all the development infrastructure we needed for server testing performance, testing integrations, all that kind of stuff. And then we had our production operations, which handled the product once we were done and we shipped it out, and they would operate it for our clients.

We had actually gotten to the point where those were separate organizations, completely separate organizations. We had a new team member join our product development operations team and he came in just preaching the importance of automation and configuration management and getting teams involved in this DevOps culture.

He and I just clicked immediately and we started working together really closely on how we can automate our build pipelines, how we can reduce our lead times on development, how do we integrate more unit testing. We have been doing—I use air quotes, “unit testing”—for a long time, but it was really integration testing. And it took hours, and it took servers being set up and data being in the system, and they failed all the time because of data conditions and conflicts between tests running at the same time. And so they really weren’t unit tests. He was able to really help me to understand the characteristics of what makes a good build pipeline, a good continuous delivery pipeline.

That really got me interested. That really got me going in thinking about it, as we started to think more and more about our product going into the cloud. We’ve traditionally been an enterprise platform—clients would download the software and run it on their own hardware, we would host it for them on our own hardware. We started thinking more and more about being a cloud platform, and how can we deploy this onto traditional cloud fabrics like an AWS platform or an OpenStack platform.

Then I got more and more involved in that from the Blackboard Learn product perspective, and seeing more and more the value of a continuous delivery environment, and those build pipelines and the quick feedback that you can get, where you’ve got this cloud fabric that can give you the quick provisioning and the resources you need to be able to deploy your product on a cloud, but then you need all that front matter. You need all that pipeline to really make it work from the point of a developer writing a piece of code and seeing it work on that cloud.

It started to really tie together that value chain from a product development perspective all the way through to that production operations perspective. Over the last year and a half working much, much more on the cloud software—and I’ll talk about more of that in the presentation at the conference—it became much more vivid about the value of DevOps, the value of build pipelines and lead times and how all that can tie together to make a much better product offering at the end for our clients.

Todd:  That was the next place I was going to go. I think that there’s so much pain in the world of IT that when we get to that conversation about DevOps, and the things that it can do, and the pain it can alleviate within the IT organization, everybody knows that and sort of feels that.

The question I was going to ask is that next step of— we always get the, “Yeah, but…” “Yeah, but what about…? What’s it going to do for the busienss? What’s it going to do? What business result is this going to— what metric is this going to drive, inside the organization that I can get people outside of IT interested in what’s going on?” How do you answer that question for folks?

David:  It’s really uniquely dependent on the company. I think DevOps, again as a culture and really more of a way of a group thinking and working together, can offer benefits to whatever the pain points are for an organization. One of the key pain points that we’ve had within the organization has been around quality.

We’ve had, over our past, several bumps along the road of our product quality, and a lot of it is attributable to the length of time our testing took and the reliability of our testing and the coverage of our testing. Something like DevOps, which reduces those lead times and really pushes you towards unit testing and quick feedback and even all the way through the operation side and how you handle monitoring and how you handle the transparency of the operational environment to your developers. All that provides a good feedback loop to your developers to provide a better quality product.

It’s now not just, “What are the bug counts?”, but how is this product really affecting both our external customers and our internal consumers in terms of our operations teams to keep this product running. You have much more visibility.

The developers, they don’t always see. They think everything is green because they got past their go/no go, and the product went out, so everything seems peachy keen. But until you get that feedback coming in through monitoring and through other feedback loops that could come in through a DevOps culture, you don’t necessarily get that feeling of, “Well, there’s really an end quality we have to worry about.”

Within Blackboard, DevOps is really … Our focus has really been about, “What can we do on our quality focus? How do we make better tests? How do we integrate those into our pipeline more efficiently? How do we get better feedback faster? How do we get the monitoring and information back to our developers so that we understand proactively where we have performance problems, or where we have stability problems,  or where we have frequent failures so we can focus our energy on improving those with every single release that we do?”

Todd:  If quality is the primary thing that you guys have been able to rally behind there, what are the ancillary things that you got as a result of it? Sort of those, “Because we put the work in here, you know what? We ended up getting all this other stuff that ended up being really helpful.” What are a couple of things that sort of come to mind to you as the extra added bonuses?

David:  From my perspective, one of the key benefits has been a better sense of ownership. Again, seeing the work that you’ve done make it all the way through the chain and getting that green light at the end of the pipeline, and seeing your features go out to the public and getting some sort of feedback.

I’ll be honest, we’re still in the midst of this. We’re still in the process of building this out. Our pipeline is not finished and we haven’t gotten to a full continuous delivery environment where our tests are getting green lights everyday and we’re getting new features out. This is our goal, our long term goal, and hopefully within this year, hopefully by the time we’re at the conference, I’ll be able to talk about it in more detail.

Even in the interim, the pieces that we’ve done so far, the level of ownership, that feeling of, “I did this work and I got the feedback that says I’m doing a good job and I’ve written something good.” It really boosts the morale and the involvement of the team in knowing that they’re doing something good for the organization and they’re having an impact on the organization.

Really, from a morale perspective and from a retention perspective—I spent some time on the management side, I still think about those people case things—keeping your team happy and making them feel like what they’re doing is having a material impact on the company’s success is really, is a great thing for a successful company.

I’ve started to see some of that loop around where people are feeling like the work that they’re doing is really impactful. Some of the other things … What else? Our pace has picked up. We’re able to get things done faster. There’s a broader sense of urgency, this desire to get things done faster and not belabor things, and think, “Oh, well, it’s okay for us to try something because we know we don’t have a long wait time before we know whether it was successful or not.”

We’ll get things out in front of people as fast as possible and get feedback and we’ll try it again. That sense of moving away from “that cliff is coming at us and we need to get it perfect before we go over the cliff or else we’re going to fall instead of fly.” We can now say, “It’s okay because it’s not a cliff, it’s just a hill,” and we can always re-calibrate and fix it before we go all the way down.

Todd:  David, I look forward to seeing you at the summit.

David:  Thanks. I’m looking forward to coming.

David Ashman is the Chief Architect of the Blackboard Learn product, Blackboard’s flagship teaching and learning platform. David is responsible for the overall technical vision and leadership of the Learn product and platform. David has led Blackboard’s recent cloud strategy, focusing on taking Learn into the cloud. David joined Blackboard in 2005.

David will be speaking at the DevOps Enterprise Summit in October.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
•