The following is an excerpt from a presentation at DOES London 2017 by Mieke Deenen, titled “Building Technical and Organizational Confidence Through Automated Deployments.”
In it she shares UWV’s journey and how they paved the way for DevOps in a Dutch government agency. You can watch the video of the presentation here.
Building Technical and Organizational Confidence Through Automated Deployments
Today, I would like to tell you about a Dutch government organization and its journey to DevOps, but first, a bit of background.
I’m an independent consultant from the Netherland. I help companies with innovative projects and right now, I am helping UWV with deployment automation.
We are a government agency that believes both people and society on a whole functions best when as many people as possible can participate in it by working. We do our best to offer people new prospects, and if work is not possible, then we ensure that income is available quickly. And that’s also our logo: working on prospects.
We are situated in Amsterdam. I have 18,000 colleagues. We execute a Dutch social and labor policy. Almost every Dutch adult is insured with us, and we have at moment 1.4 million benefit recipients — people that rely on us.
Our clients are a very vulnerable group. Usually, they are people who don’t have much in savings, they live paycheck to paycheck, and they are not on our website because they want to be — they are on our website because they have to be. When we fail, when we are one day late with our payments, we get these people in serious trouble. They might not be able to buy their groceries or pay their bills, etc. So every time we have even a little mistake, it’s a serious problem for the people who depend on us.
From an IT perspective, we have a large enterprise architecture, about 300 applications. The front side is quite modern, but the back side is a bit older. The oldest application we are currently running is from the 1960’s. We are a siloed organization. We have our development outsourced. We have all the work on acceptance and production outsourced with another hosting party. We are a process-oriented organization, and we are risk-averse. Our three main KPIs are stability, availability, and performance.
a bit of our history
In 2012, new legislation was passed, and we had to digitize all our services. For example — we had a static website with some basic information, but we needed to create a dynamic website with self-service portals for employers, employees, for our business partners. And we only had a couple of months to do this.
We managed to do that, but then we suffered some small crashes, and we asked an external company to audit our website. They wrote a report, which made the reality very, very clear they said, “Your website is not future-proof.”
So the next year, we put all our effort into increasing the stability of our website. Nonetheless, in the summer of 2013, we suffered another major crash. It lasted for several days. It was big national news. It was on television every night. It was in all the newspapers and is probably the worst that had happened in our history.
The result of this was a very tightened the process. If you wanted to change something on production, you had to register your request for change 42-days in advance. There were several quality gates. There was a freeze period at the end where there was some testing, and you had to pass this whole process in order to get something into production.
It became so difficult to pass the process, that we had almost no releases anymore. It was once or twice a year, maybe if you were really good at it, three times a year — but that was it. That created more technical debt, more problems, increased costs, and people had lost their faith in innovation.
So we figured, “Well, this is not what we wanted. We have to find something different. How can we increase the quality? How can we increase the availability of our applications? How can we provide new functionality to our customers and help them, because they rely on us.”
And then, of course, we came with the ultimate question…
Can our organization even use DevOps?
This is actually when I started to work there. We started very small with a proof of concept. We built a sandbox in a test environment, and we had a small application with low impact. Our focus was mainly technical. We just wanted to answer the question: “Can we do this?” And as you might not be surprised to learn, we did. We actually doubled the time that it usually took to deploy the application, but we had the technical proof that we needed, and the extra time was also partially because we had some extra tests and checks in there too, so it was okay.
From here, we moved on to start a real project. We started with one division, customer service. Customer services has 30 applications. I wrote a project initiation document and a business case, with the costs and the benefits, and then asked for permission to start, and we started it.
When you start to implement a DevOps project, there are always some challenges that you will have to face, that every company will face. There are also some challenges that we faced that are typical for a governmental organization or an organization is siloed. It’s these challenges that I would like to share with you, and also how we managed to get through them.
The First Challenge
Like I mentioned earlier, we are a siloed organization. We had our developers on one side. We were in the middle, and our hosting partner was at the end. People were not supposed to talk to each other. Developers weren’t supposed to talk to people who were working operations or production. It was very strictly separated.
I figured, “Well, “I cannot remove the walls. I cannot change the contract. But I can show them what is behind the wall.” I invited them all to come and work together. I showed them who the next person in the chain would be and showed them what their work was. This is how we started to communicate. Of course, this sounds very easy, but it definitely wasn’t. At first, they just felt like they were forced to work at our place instead of with their own colleagues. It was a start.
But they weren’t a team yet. In my vision, I wanted a team, a DevOps team with Agile values. So I made a Scrum Board and started daily stand-ups. That was also very unusual to them, because they were used not to sharing any details, especially not when something went wrong. But I wanted transparency. I wanted them to share everything.
In turn, I created a place where it was safe to talk and to share, and it created more focus, more commitment. We realized that we were all working on the same things. It wouldn’t help if we blamed each other, we just needed to help each other.
The Second Challenge
The next challenge was the process. I figured, “Well, I cannot register a request for change 42 days in advance. I don’t even know all the details yet, but more importantly… I don’t want to.” I felt like, “This is the complete opposite of what we are trying to achieve. It has nothing to do with making things easier.”
So I invited the Change Manager for a cup of coffee. I told him that I was going to implement a new way of working and deployment automation and I would like him to adjust the change process. And he just smiled, and he said, “Lady if your way of working is not fitting with this process, you have to adjust your way to working.”
Well, we had another coffee. I told him, “We’re both working for the same company. We’re even both working for IT. We have the same CIO. This is my assignment, and that is your assignment. But somehow, our CIO seems to think that we are able to work together and that we are both able to fulfill our jobs. So we have to work this out.” So we agreed to disagree for a while and then started to work together.
He showed me everything about the process, and he explained the purpose of the quality gates and what was really important to him about them. I showed him more about our way of working. I could also show him that we would actually meet all the acceptance criteria of his quality gates, it was just done in another way without so much upfront.
Eventually, I gained his trust, and he became an ambassador. He helped me through the process and through the bureaucracy. Most importantly, after a while, he even gave us the permission to register a change only five days upfront — which made things a lot easier.
The Third Challenge
Our next challenge was with our hosting partner. We pay them on a time-material basis, so every hour they spend on deployments is revenue for them.
I had to ask them to help me automate their work, and they, of course, assumed that I was asking them to cut their profits.
They perceived me as sort of a cowboy, someone who was not willing to follow the process. Which was very difficult at first as I had many, many impediments and challenges to move through because of this. But after a while, we realized that we actually had the same goal. We both wanted quality. We both wanted availability. We both wanted stability.
So I started to ask them more for their opinions, and they started to give me more advice. Until eventually we began to behave more like partners. Which made it a lot easier, and they became part of the success. They were helping us to get to DevOps.
The Fourth Challenge
Another challenge we faced was staying clear on our goals, even when we lost them because there were so many impediments.
I figured, “You have to visualize your goal and tackle every impediment, one by one.” We learned that an impediment is not a showstopper. Once you tackled it, you’re really one step closer to your goal. So we had to be very patient and very persistent heading towards our goal.
After about six months, we were finally allowed to deploy on Acceptance, which went well. Then I wanted to go on to production, which would be a new moment for some new challenges, and I had to work out a new strategy.
Usually, when you want to deploy on production, you announce it months before. You do it in a weekend or if it’s a small application, in the evening hours.
I didn’t do that. I talked to my Change Manager and said, “Is it alright if we just prepare it and do it. I’ll take full responsibility.” And he agreed.
Then we prepared the deployment, and planned it for a very, very, busy office day. I waited until it was about time to go to lunch and I wrote an email to my stakeholders. I said, “Hi, everybody, we are going to deploy in production. If you do have any questions, please let me know.” I pressed send, and we deployed.
We tested it, and everything went right. So I wrote another email. “We did it, everything worked, thank you. Bye.” By the time most of the people realized what we were doing, we had already done it. And although not everybody was very happy with the way of announcing it, they were also a bit proud. We gained some support, some respect, and some confidence. And it was a new way of working, but from then on, it went fast.
During the summer months, we did all the other applications from the Customer Services. We onboarded them one-by-one. By the time everybody came back their vacation, deployment automation was business as usual for Customer Services.
Here were our results
We improved the quality and the stability. They now have bi-weekly releases. We even over-achieved the financial benefits, although that was never our main goal. Our main goal was the quality. And we gained a lot of trust. We built confidence in innovation. We built the confidence to change.
I had to present my results for my end-of-project to the Executive Board, and they were very happy. They made an important decision: They said that it would become policy that all applications should use deployment automation, for all the applications, not only for Customer Services.
This is where I am right now. I’m building a factory because we have to scale. We have to go to 300 applications now. We have a whole bunch of new technologies, which are actually the old technologies, and we want to make a pipeline where we can onboard every application. Where we can adjust, where we can experiment.
We’re going to finish the bridge to the future, so that everyone, every old application, every modern application, everyone can onboard, and everyone can cross the paved road to DevOps.
My Final Takeaways
Maybe your organization is considering DevOps. My advice would be, just do it. Start very small, start with a small team, with motivated and skilled individuals.
Dare to be different. Dare to work in different ways than the rest of your organization. Try DevOps, even if the rest of your organization isn’t ready for it yet.
Build trust. Create bridges, bridges to people, but also to the future. Make the goal visible, and be very patient. There is no innovation without resistance, so you have to be very patient working towards your goals. Keep learning and improving. Look for opportunities.
There is no moment waiting for you. You just have to grab the moment. Celebrate your success and do it again.