How DevOps Can Fix Federal Government IT

When thinking about federal government IT, there are two words that seem to never go well with each other– bureaucracy and DevOps. Actually, they’re almost complete opposites. Bureaucracy is inherently slow and difficult to make significant change within, while DevOps is everything but!

Smaller, newer companies are able to implement DevOps practices more easily because they are nimble, and the decision making power lies with small groups or even individuals.

[bctt tweet=”Smaller, newer companies are able to implement DevOps practices more easily because they are nimble.” username=”DOES_USA”]

While it may seem crazy, today I wanted to have a conversation about bureaucracy in a way that isn’t negative, yet deals with the largest, most bureaucratic agency of all- the Federal Government.  

I joined the federal government for the first time just over 4 years ago.

I was coming from the Bay area startup community, and knew it was going to be somewhat different. For one thing people called me sir, even when they weren’t being sarcastic.

Another big change was that I went from an environment where I was managing 30 people to managing about 2,000 people. From a budget of about 3 million dollars a year to about 500 million dollars a year. Clearly, these were some big changes for me.

The government has a particular culture of bureaucracy because of its very nature.

It’s run by rules rather than arbitrariness, which is really the original definition of bureaucracy.

Everybody is always talking about the bloat in the bureaucracy, so when I first joined the government the thought in my head was, can we have a lean bureaucracy?

[bctt tweet=”The government has a particular culture of bureaucracy because of its very nature.” username=”DOES_USA”]

Can we make it agile, is that even possible?

I set out to try, and I was met with some interesting results.

One of the clearest examples was in a big conference room, filled with people. Most government meetings have a lot of people. It’s kind of a rule.

We’re discussing a legacy system that nobody was using anymore and it was costing us a lot of money to maintain, called ARNAX. With a name like ARNAX  it has to be a legacy system right?

Since nobody was using it and I’m the new CIO I say, ‘All right, let’s decommission it.’

It goes totally silent in the room and just like that, I know I’ve said something wrong. A little voice from the back of the room says, ‘Sir, you don’t have the authority to do that. It’s in the MD102.’

MD102, the epitome of bureaucracy.

It stands for Management Directive number 102. Our agency is a part of the department of Homeland Security, and they were the ones to write it. They had sent me a copy of MD102 before I started working and said I should probably read it.

I glanced through it and realized immediately it had nothing to do with the way we actually develop software, so I didn’t bother. In retrospect that might have been a little bit of a mistake. I did actually have to read it.

But, I didn’t want to just read it, my copy is highlighted, underlined, it’s even got little comments in the margins.

I studied every word of it and I became the greatest expert in the government about MD102, and what I found might surprise you.

While bloated and superfluous, it’s a beautiful document, absolutely brilliant for instructing a large group of people using the waterfall approach.

For instance, it says, when you’re developing software you have to divide everything into 9 phases, you have to have a solution engineering phase, a requirements phase, a design phase and so on.

Then, there’s a gate review during each of those phases, and 11 gate reviews all together. My favorite was the ‘Test Readiness Review’ which makes sure that you’ve finished development before you can start testing.

Clearly MD102 was the enemy to achieving lean bureaucracy, and I knew I had to fight it.

So what did I do? I brought in a whole bunch of agile coaches to help make up my own policy, and called it MD001. It said that from now on we were going to be agile. I figured I’d just try and see what would happen.

Well, I rolled it out and you won’t believe how well it actually worked!

MD001 defined agility in a very reasonable way and I got myself appointed to a group that was going to rewrite and improve MD102!

Within a year we were really starting to move.

We had done a hundred agile releases, we had great product ownership from the business side, and everything was going fine. That is until one day as I was rewriting MD102, and I had this weird epiphany. I started to think about MD102 in a very different way.

I had just sat down with the people who had written it and I realized what had seemed like outright horrible, faceless bureaucracy is actually human beings in a government organization trying to do the right thing- Only without knowing a thing about IT!

So I went back to the MD102 and I began to see what a tremendously human document this was. I know that’s a little strange to say that when you’re talking about Test Readiness Reviews, but if you read it the right way you could feel the fears of these people who are writing the document. The hopes, the dreams, the integrity, the commitment.

What I realized was that in order to bring agile practice into the government what we’d have to do is somehow meet the needs that are expressed in MD102, and instead of trying to undo it all together, find a better solution to meeting those needs.

But there was another layer.

If you think about how our government is structured, we have a system of checks and balances. We have a low trust environment with congressional oversight looking over our shoulders all the time, the press is looking at what we do, the public is looking at what we do critically.

Could we find an agile, lean practice that will satisfy all those things as well?

It turns out that yes, we could.

DevOps was created to solve this need.

We started tooling up, we have three different big projects going with continuous delivery pipelines. We have a developer working on JavaScript, Hibernate sort of stack and some working in Ruby on Rails. We have automated testing, continuous integration, continuous delivery into the Amazon Cloud.

Where we were using a scrum based process before, we’re now using more of a DevOps based approach.

Let me give you an example: one thing about the federal government is that there are very strict procurement laws.

What might be a good strategy for some companies is something we legally have to do.

For instance, it’s very common for us to have to change contractors over time. It’s important that we create a level playing field for each different contractor, and one way we’ve dealt with that is to create a lot of documentation.

This is horribly consuming, but in a DevOps practice with a continuous delivery practice, everything’s scripted. There’s isn’t any documentation to be written for the next contractor. The day they start they can deploy to production, the test suite, the regression test suite and get working right away. Re-factor, make changes, mess things up, it doesn’t matter. A well thought out DevOps practice helps support the fact that we have lots of contractors coming in and out.

Here’s another example: compliance.

The government runs on absolute compliance. So one of the things that we try to do is turn anything we have to comply with into an automated test. If we have automated test suites that check for FISMA (Federal Information Security Management Act) then compliance becomes a matter of doing exactly what the tests are testing for.

By taking the compliance issues and incorporating them into automated tests, we know our system is always deployable, we’re not going to find out at the end that it’s out of compliance with a well defined problem. As long as it meets those tests it’s compliant by definition.

DevOps gives us the potential to take FISMA compliance to the next level, and I think we can improve the government security posture considerably.

Another change that was already going on in the government before we started to introduce DevOps was continuous monitoring of production to find vulnerabilities.

But now instead of checking compliance of a system when we release it and then every two or three years, we have ongoing authorization and ongoing monitoring in production to look for vulnerabilities.

With a DevOps approach, we can implement production for feedback with continuous monitoring tools along with our performance tools. We can have constant feedback to the developers on how secure the system is or what vulnerabilities are found.

[bctt tweet=”With a DevOps approach, we can implement production for feedback with continuous monitoring tools along with our performance tools.” username=”DOES_USA”]

If there’s a problem we patch it, push the button, deploy a new version. In fact in the Cloud we can tear down the old version that might have been compromised and spin up a new version.

These are just a few ways that the government has very special needs that can still be met with agile practice.

And the result is that people have begun to do things different, but not in a way that rips out the cultural foundations of the work place. Even in a deliberately a low trust environment, the best way to solve it’s problems is still an agile way, and a DevOps process, in particular, maps very nicely into all of government’s processes.

- About The Authors

Mark Schwartz

Mark Schwartz is an iconoclastic CIO and a playful crafter of ideas, an inveterate purveyor of lucubratory prose. He has been an IT leader in organizations small and large, public, private, and nonprofit. As an Enterprise Strategist for Amazon Web Services, he uses his CIO experience to bring strategies to enterprises or enterprises to strategies, and bring both to the cloud. As the CIO of US Citizenship and Immigration Services, he provoked the federal government into adopting Agile and DevOps practices. He is pretty sure that when he was the CIO of Intrax Cultural Exchange he was the first person ever to use business intelligence and supply chain analytics to place au pairs with the right host families. Mark speaks frequently on innovation, bureaucratic implications of DevOps, and using Agile processes in low-trust environments. With a BS in computer science from Yale, a master’s in philosophy from Yale, and an MBA from Wharton, Mark is either an expert on the business value of IT or else he just thinks about it a lot.

Follow Mark on Social Media

• 
•