Scaling Continuous Delivery at Walmart

But if we don’t talk about DevOps, what do we talk about? What we do instead is focus in on the outcomes that we’re looking for and foster the culture to attain them. We’re all here looking to deliver quality software rapidly. And the key to the culture change that’s needed to attain that outcome is our people and our teams.

When the team is able to solve those problems, they not only become good problem solvers, but it generates a lot of teamwork. You get a team that can deliver value very rapidly. For instance, the team that I came from went from 0-12 deliveries a day to production.

I went to one a couple of years ago and I was also very excited to bring continuous delivery back to my team. I knew that it would make our lives easier. I knew that it would work better with our business partners and deliver value faster.

The problem I encountered was that I couldn’t find an area within the organization to learn more, to find out what initiatives we’re currently working on and how to implement continuous delivery.

I looked around and found a lot of pockets of really good progress. We had teams that were building pipelines, we had teams that were focused on testing and continuous integration, and many teams were trying to solve exactly the same problems independently.

I had to figure this out, so I decided to host another DevOps day and I brought in leaders and developers from all over the country to share the vision and highlight the progress that they were making within the organization.

Unfortunately, I knew that this event was going to garner a whole lot more excitement and that people were going to learn more, and want to accelerate faster. But these same excited people were going to end up just like I did, looking for that central area to kind of guide them and what those next steps are.

So, I built us a home. I started Continuous Chai, which is a CI/CD user group where people can come to share and learn about continuous integration, continuous delivery, and the myriad of topics that go along with it.

This community of sharing is the first of four initiatives that we want to share with you today.

I believe that when we want to change culture, it works to help use that culture to teach the culture.

Sharing is a key tenant of DevOps and it’s important to share early and often. We have old habits and human nature that hold us back. We only want to show people beautiful, shiny, finished products after we’re all done. Then we say, “Look how successful I was.”

What we’ve built in Continuous Chai is a forum where people have the freedom to share off the cuff ideas, to share their work in progress, and to highlight not just their successes, but a trusting environment where they can honestly speak about their failures.

By sharing early and openly, teams can learn a lot from each other and avoid wasting time with duplicate work efforts struggling to solve the same problems alone. This act of trusting user community is key to enabling large scale change.

And we said, “Well, have you asked in Continuous Chai?” And they go to the community and you have the community dive in and help them. You get solutions so much faster you do by Google or Stack Overflow.

In fact, when Dana and I were working on this stack together, we went to Continuous Chai to get feedback because we knew not only that we had a trusting environment with friends, but we knew we were getting actual real feedback, not, “Oh yes, it’s wonderful.” It was a little daunting to see all the notes come by, but it absolutely helped us improve this material.

But, I’ve also learned some things along the way. First of all, a leader of a user community has to have passion. It’s not something that you can just tell someone to take care of and expect it to be finished. Building a community takes ongoing work to engage associates, maintaining a consistent schedule, and bringing interesting demos and discussion topics to the group.

Secondly, it takes a lot of patience. When we first started, there were many times when I was sitting in a room by myself or with one or two people. Yet, even a handful of people can brainstorm ideas and start bringing true value to the group. We have iterated in different formats along the way. We’ve done informal coffee chats, we’ve done demo and discussions focused on specific topics, and even offsite meetings.

Over time, we’ve come to find that we have the most success in our environment with meetings that have specific demo topics. We just keep iterating on the format and on the timing, and we currently have over 600 members and offer weekly demo and discussion sessions. It can’t be built in a day. When momentum does periodically slow because it will, there’s one fail-safe way to incentivize people to keep showing up — swag and free food.

The other thing is a unified platform.

When we first started on this journey, we had several areas and the areas were really digging into CD, they’d go and spin up their own Jenkins instance or whatever tooling they were using while other areas weren’t focusing on it all. They didn’t have the tooling, and didn’t have the bandwidth to get it done.

But it doesn’t scale for every single area, or every single team to get their own platform stood up. Product teams should be focusing on delivering products. Having a consistent unified platform they can use and is easy is absolutely key.

I work in software delivery enablement where the area that’s responsible for building the CD platform and it’s a set of tools we’re building. It’s delivery as a service. We want teams to be able to focus on those products and then just use the automation to deliver. We don’t deliver it for them. We just build automation.

We’re using open source tools and scaling them to Walmart. And I’ll tell you, we break a lot of tools, and we want to make the right thing the easy thing, we want you to flow downhill to success. The initiative we work on is called Irresistible Developer Experience. We want you to use the tools because they believe that they are better, that they’re easier to use and we find it’s really fast on more people on these tools.

It has also been planned from day one to release this back to the community as open source, which has enabled a good use case for Dana’s team.

We’re able to implement our enterprise platform in our segregated network and very easily pull in the new features and still be able to take advantage of all the work going on across the enterprise.

Here is an example of how simple it is to configure the tools from the developer’s standpoint.

We simply have a configuration file located right alongside the code using a simple declarative language. This allows for configuring and versioning individual repositories to be very simple.

It hides the complexity from the developers, and each feature is just a simple function call. We can see right that a single line of code calls Hygieia and publishes the build metrics from this repo.

Therefore, it’s important to make those goals clear and the metrics clear so they understand how they’re trending against those goals.

To do that we use Hygieia. If you don’t know about Hygieia, Capitol One open-sourced Hygieia several years ago. Teams around our building had been using it for years, and we’ve now we’ve integrated into our pipeline. It gives you a real-time view of the CD pipelines and is a really important tool for the teams.

We’ve also added scoring to Hygieia. The metrics are weighted and aggregated to give an overall health score. This scoring gamification helps drive improvement and it allows teams to quickly see which code bases are more hardened and which might need a little bit of attention.

Teams can analyze where they might need to put attention by drilling down into each individual metric widget. For example, we can see that most of the code repo score is determined by the frequency of merge to master, but if teams are committing directly to master without using a pull request, then they’re going to take a hit on that score.

Metrics are incredibly dangerous things if used inappropriately. You need to really understand those metrics to understand how people react to those metrics. Just because you put a metric in place and expect an outcome doesn’t mean you’re going to get it. Go and investigate.

An example we had of this, was when we implemented the scoring and we had teams coming to us saying, “Now I have to go and make changes to repositories that currently don’t need changes, just to keep the score up.” Well, that just generates waste. We don’t like waste and we want value.

We made some additional changes to make things better. We created a higher level view on top of Hygieia that aggregates the metrics up and averages them across the team.

We have a tool in inside Walmart that tells us how big a development team is, how many engineers are on that team. Therefore, we can average the scores based off of the team size and we can get those deploys per day, per developer, or commits to master per day and find out how teams are doing that to say, “Here are our goals, here’s where you are, how do we help you achieve those goals?”

But even then, we currently have all the scores weighted equally even though commits and deploys are far more important than code coverage. We have teams that are right now trying to raise their scores by raising your code coverage, which was incredibly easy to do.

What we’ll do in the near future is to drop the waiting on code coverage and increase the waiting on commits and deploy to get outcomes we want.

Furthermore, all of the scoring and the widget changes, we have pushed those back to Capital One and you can find those on their master today.

While Hygeia does give us the metrics to evaluate how we’re doing and as a team where improvements might be needed, sometimes teams need a little extra help to actually determine how to implement those next steps and gets to the next level.

We’re a group of developers who’ve done this before and we can embed with teams and help them out. The team that I lead is CD Sherpa Team. We have been up and down the mountain. We know where the ravines are, we know where the landmarks are, we don’t want you to become one the landmarks so we help with anything required to get it done.

We do platform support for the tools to make sure you understand how to use the tools, but we were also run tech workshops on domain driven design or my favorite one, Agile rehab. That was a really popular one.

We do leadership outreach where we explain that this is a change in how teams should work. You need to understand how this impacts how you incentivize the teams to get the outcomes you want.

We do team boot camps where we will embed with the team for six weeks run two and a half day sprints. It’s very similar to the other dojo’s you may have heard about from other companies, and help the team with whatever their biggest constraint is helping move the needle, show them that improvements not only possible, but it can be really fun and build teamwork.

Finally, we are pie-shaped developers, not T-shaped developers. We have to have breadth and depth and it’s really hard to find people for this team. You can talk to anybody trying to build teams like this. It’s incredibly difficult.

We tell the teams, we’re not Agile coaches. We can coach you on Agile because you have to be good on this stuff to get this done. But we will also help you with planning out of legacy strangulation. If you need help with test architecture. We’ll peer program with you to teach you how to unit test if you need, we’ll do anything required.

If our team doesn’t know, we’ve been here for a long time, we know people that do, we’ll bring them in and get that knowledge to you as fast as we can.

Growing an excited base of people to advocate every day is important. We’re reaching out to leadership and developing a strategy that includes a single enterprise deployment pipeline, metrics that are focused on the right outcomes, and teams dedicated to training and enabling to deliver value safely and quickly.

The single pipeline across all teams and tech stacks makes the right thing, the easy thing to do. Standardized metrics make progress visible and understandable at different levels because it’s standardized. The gamification makes it competitive and fun. Our people really keep the momentum going to enable large scale change.

We’re seeing a lot of improvement using this process, but context is really important. You need to u understand that nothing you see is a cookie cutter solution. You need to find out what works in your culture. If people were incentivized by badges, give them badges. If they’re incentivized by certifications, do that. Whatever it takes to move the needle, get it done.

Also, it’s important to understand, you need to give people permission. Dana and I are not management. We are developers. Walmart has a strong culture of grassroots improvement. We didn’t ask for permission. When Dana needed a DevOps Day to learn more. She said, “How do I reserve the auditorium?” Not “May I have a DevOps day?”

When she decided to start Continuous Chai, she just got meeting rooms together, spun up a Slack channel, started Continuous Chai, and then made all of the leadership help.

There are people that are passionate about this in your organization, so make sure they know they have permission. Don’t assume. Find those people, elevate them and give them all the runway to bring everybody else along.

First of all, teams are collaborating. Lots of collaboration between teams is helping to remove duplication of efforts and to shorten the learning curve for teams. Teams who were focused on continuous delivery are delivering faster and they’re delivering with higher quality. When they see that and start to do that, they realize that CD removes the drama from delivery and improvement is addictive. Teams are actively trying to improve and using metrics to measure that progress.

Thanks very much.