The following is an excerpt from a presentation by Dan Cundiff (Principal Engineer,) Levi Geinert (Director of Engineering,) and Lucas Rettig (Principal Product Owner) at Target, titled “Crowdsourcing Technology Governance.”
Slides visuals by Nicole Forsythe (@nicolibrarian on Twitter.)
You can watch the video of the presentation, which was originally delivered at the 2018 DevOps Enterprise Summit in Las Vegas.
A Little Bit About Me.
I’m a director of engineering in the Target Dojo, where we help teams with guidance on engineering choices as well as changing the way they work.
I’m going to share a little bit about how we moved from a governance model to a guidance model and how that’s helped support the empowerment of our engineers and our business to make smart decisions, take smart risks, and be empowered.
We’ll also give a real example of how we’ve implemented this by crowdsourcing some technology guidance.
A Little Bit About Target.
We’re big. We have a lot of stores. We give back to our community in many ways. The interesting thing to note here is that we have a lot of team members. Over 300 product teams as well as 4,500 people plus in technology. This is why guidance is important to us.
A little bit about our evolution.
We started our journey with DevOps by rebuilding our engineering culture. Which was grounded in the empowerment of our engineers, DevOps, agile, and so forth. This had a very big impact on our engineering capabilities at Target.
After the DevOps Movement, if you will, we got into changing the way we work. The Dojo has supported this. Much of our business today is from business teams that want to change the way they work. They’re seeing how our engineering teams are empowered and taking risks, and they have an interest in that. They also want to understand how to interact better with their engineering teams that are operating with Agile, Scrum, or Kanban.
Finally, we realized that to have great products, we needed to have our customer at the table. I’ll show you a tweet that’s one of my favorites. Kaitlin mentioned that “It sucks when you have an idea for a product, but it’s much worse when you build the wrong product.” Aligning on our customers internal or external was very important for us, to build those awesome products.
Governance to Guidance
On to governance and why that was something that we wanted to get out of the way.
We didn’t want to tell teams how they should work, how they should operate, what tools they should use, because when we empower them and we make them accountable. Then they choose the right tools. We know this because we have research to support it. And I would extend this to not just tools, but languages, frameworks, and all the choices that teams need to make to accomplish and build those awesome products.
Now, it’s great to remove governance, but what happens when you don’t provide guidance? We know that it can get very noisy in a tech landscape. If you do any searching of images around tool choice, it can be overwhelming. Teams that are given no guidance have to consider every possible option. Guidance allows us to streamline and reduce the friction required to make those choices.
What Does Guidance Look Like?
We’ll go over some examples, but let’s first discuss some of the principles.
- We believe that guidance should be accessible, and available to anyone to contribute. Not a specific leader, but anyone at any level.
- We think it should be transparent. We hope that anyone can view it. It’s not a great situation if the engineers aren’t able to find out what are the best tools that they should be using.
- It needs to be able to change. We know that this stuff changes very quickly in our industry: tools, languages, frameworks, etc.
- Finally, guidance is cultural. Just like DevOps, our guidance model needs to be cultural. The people that are providing the guidance also need to be supported and able to promote other leaders to step up.
In many cases, our Dojo is not the expert. We may have an awareness, however, of what people are doing, or what patterns are being used, which means we’re able to bring in people to help those teams with a specific technology or framework.
The Dojo also is a part of guidance. It’s a big part of our culture. It helps us empower teams, and support them to take smart risks. I want to announce that we’ve recently had over 350 engagements in our Dojo, 250 of those are 30-day challenges. We’ve had over 180 tours of our Dojo from many other companies, as well as internal teams. Again, trying to help guide people towards positive change through empowerment.
I’m a principal engineer at Target Corporation. What I want to share with you is a real-world example of is how we take this idea of empowering engineers and the idea of going from governance to guidance and put it into action.
First, I want to set the stage for how we did things before.
This might be a familiar process at your company. Before, we had a centralized group that would choose technologies for us. We called it an Architectural Review Board, an ARB.
What’s difficult about these centralized groups is that they have to make decisions for many different product teams, and the larger the company you have, the more likely they are to pick the technology that doesn’t work for everyone or take into account all the different product teams’ unique things they need.
That becomes an issue.
Secondly, if you are someone who wants to be able to influence that ARB or that centralized group that’s doing that governance, it’s probably going to be pretty hard to do. Usually, you have to go through their process, they probably meet on a regular basis, they have some form you got to fill out, etc, and these things are problems.
ATTEND THE DEVOPS ENTERPRISE SUMMIT
Here’s what we did— about four years ago, we got together and we’re like, “Hey, this can be done better. Let’s try and disrupt this. Can we do this in a different way?”
So, we created a repo in GitHub of a simple list of technology choices, that we wanted to use as a stage to make a couple of new disruptive choices and to see if this thing could take off.
This is the repo.
It used to be a single file, broken up into different categories. What you see in this list are different sets of technology categories. Collaboration tools, application frameworks, caching, data stores, etc.
You can go into any one of these files, for example, let’s take a look at the data stores ones. In there is typically some front matter on the file, for instance, an introduction, some motivations for why people maintain this file, and why they make the choices that they do.
If you’re a developer and you’re coming here saying, “Hey, what choices can I make or how can I influence this document?” They can scroll down a little bit further to a different section, to see what the technologies are and what their status is.
Let’s say you’re looking for a relational database store and you’re wanting to see what your options are. Post Press is a choice here. The other fields to the right, the use case etc, that’s just more like metadata. The real important one is the status.
There’s recommended, there’s limited use, and do not use. If you do try to do this yourself, you might feel tempted to have all kinds of other statuses, but keep it simple.
‘Recommended’ means you can use it.
If it’s ‘limited use,’ there’s probably some reason why, maybe there’s some licensing constraints to it, costs, maybe it doesn’t implement well in certain environments inside of Target.
Then the last one is, ‘do not use.’ You shouldn’t use that here at Target. In fact, we even put things on this list that we’ve never used at Target just to make sure someone doesn’t think they can go choose it. We’ll also say why it’s not a good fit.
Let’s talk about the motivations for why we did this.
Your developers are already in GitHub day to day. They’re writing code, they’re doing social coding etc. If you decide to do something like this and you empowered your engineers, you can put it somewhere where they’re already working. Put it in version control. There’re a lot of benefits to that.
Pull Request Workflow
The pull request workflow is ‘I have a change and I am requesting that you pull in my work.’ Anybody at Target, you can be a junior engineer, a more senior person, can open up a pull request to any of these technology categories and suggest a change.
We’ve had junior engineers do this who they just maybe have learned something new and they want to bring it forward and say, “Hey, can we start using this? I think this is a gap in our technology set.”
Some people have come forward and had to make very large changes like we need to change an operating system or a much bigger technology choice that requires a pretty big discussion.
Either way, you can do this through pull request. It’s a perfect workflow. Everyone can comment and have a discussion on it, and when it gets merged, then that’s it. The technology choice is final.
Another nice benefit is, if you’re putting in Git, you can go to any one of these markdown files and run Git Blame on it and see why it changed.
This example up here, this is for Kafka Streams. I can see the actual commit that was done, I can go back and see the pull request and any associated issues with it. For posterity, I have everything captured to understand, say a year later, why did we make this choice. It’s all just right there in history in Git.
Here’s an example pull request.
This is a person named Dan Moss. He is suggesting a new database technology. You can see his justification up there.
He’s made a case for it, a sort of pitch. It started a discussion, and this was only a couple of days old and it already had a lot of comments on it. This is exactly how it takes place.
One note, there will be a whole spectrum of people who will chime in. Then there are also some people who want to make comments, others who just want to listen and will react with emojis. You can see that there are three thumbs up.
There are some comments in some of these pull requests where a lot of people will add all kinds of different input, whether it’s comments or different emoji to kind of show their support or which way they’re leaning.
Because it’s in Git, we can do all kinds of other things, too. Just like you are working on a project and you’re putting code in there and it can trigger workflows, it’s good to point out that there are other things that you can do as having an interface into a process like this. For example, you can have the alerts sent to slack. Or you could have a job kickoff which renders these in PDFs which can be more digestible for certain people. The good thing is that you can stitch all kinds of alerts and nice things to workflows.
ATTEND THE DEVOPS ENTERPRISE SUMMIT
After having seen this exist for four years, one important thing I want to bring up is to resist complexity.
There is going to be a real temptation to say, “Hey, we could add more status categories.” or “We could do this process to this.” “We could really require all these additional steps,” etc. Whatever it is, don’t do that. At the end of the day, it is developers who are coming here and recommending these changes. If you make it too difficult to do, they’re not going to come here and do this. Absolutely resist it.
For example, one change that was helpful was we used to have that single large file, which was nice to just do control+F on, but it got pretty cumbersome to vertically scroll through, so we did split it up.
Other helpful things that people have added have been like decorating some of these pages with motivations and the reasons why we’ve made specific choices or to give more clarity on things.
Other than that, it’s always stayed pretty simple. It’s just a Git repo. You can open up a pull request, you make your pitch, and if it gets merged, it goes.
In the earliest days, it was volunteer-led, but one piece of advice I have is to make sure that over time, you keep new people coming into it. Keep things fresh. You wouldn’t want any one person to have a real dominant voice. If someone’s going to suggest a change to this, you don’t want them to feel like they’re coming into a conversation with bias, for example. People will probably emerge and come forward, and they’ll start doing it anyway and you’ll want to embrace them.
Big Change? Take It to the Top
Sometimes, here are some changes that have a very steep cost associated with them. Or maybe the switching cost of moving from one database is high. Maybe there’s just a big disagreement that can’t be resolved in the pull request commentary. There is another part of this process where we can actually take it all the way up to the CIO.
Our CIO has a near-quarterly setting where, if you’ve made a pull request on something and it requires the CIO level input, you can pitch it to them. If you’re a junior engineer or whoever, you can make your pitch for that change to the CIO, his DRs and principal engineers and other influencers. In most cases, you’re kind of rehashing the existing conversation. There’s a set of unbiased criteria that is typically followed in that setting, while you’re making your pitch just like if you were in a startup.
The nice thing is that, not only can people come to that setting and get that type of attention, but the trust goes the other way. Our leaders at the top entrust our empowered engineers to let this guidance based process actually work — they trust this process. They already take into consideration all the input that was already had with the pull request conversation beforehand, so there’s a lot of trust built into what dialog has happened so far, and they are taking this engineer’s input into this CIO level setting.
Monitoring the Health of Your Community
Finally, measure the health of your community. See how many watchers there are getting alerts for things starred and forked. Get an understanding of how many people are actually making changes and things like that.
What I really want to underscore here though, is that just because it’s a not centralized with governance, that it’s guidance, doesn’t mean you shouldn’t measure it. You should measure to make sure everything is healthy and that things are on track, and that there’s some good vitality in the community.
I’m Luke Rettig. I’m a principal product owner at Target, I have our Enterprise GraphQL Platform, one of our Enterprise Caching Platforms, and then I have our core item location data service that we provide out to the enterprise.
I’d like to share today about my view on why DevOps, and particularly this theme of engineering empowerment, and then also product team empowerment, and how some of the things Levi touched on can really hit home and really drive awesome products.
I think we all know this, but building the wrong thing is a nightmare. It’s a nightmare from the, you lose time, you have unhappy users, you waste a ton of money and engineering cycles doing this. Then, more times than not you end up rebuilding it. And I think that building the wrong thing actually stems from two different problems.
The first problem is, if your engineers or your product development team is constrained, you’re dead in the water. I call this my square peg/round hole example. I’ve worked on so many products where there was a package implementation where I had a feature or a particular experience where we couldn’t deliver it how we wanted to deliver it. Which meant we ended up having to do a process workaround or something like 10 clicks to get through this workflow, etc. I think that having a development team that is empowered to make their choices about their technology is going to ultimately get you to a better result, or at least give you a fighting chance.
Secondly, I think that the other symptom goes all the way back to the roots of the DevOps movement to begin with, and that’s silos.
I’ve worked on many teams that have this siloed approach. If you have separate technology, product roadmaps or objectives, key results, etc. it’s going to drive this difference in silos. Your engineering team is going to want to do something that their organizational leadership is driving, while your product team is doing something else. Your design team is doing something else.
I think another characteristic is in the language you use, for example, if you’re saying terms like “the business” verses “our business” or “our users,” that’s pretty telling that you have some silos.
Another thing I hear often, that is near and dear to a lot of people is, “My product owner only prioritizes features and functions. They don’t care about debt or defects or any of those sorts of things.” I think ultimately, this leads to bad things.
I think really where we need to go to get awesome products is to have shared goals. You have to have an empowered team. You have to be on the same page. You have to break down some of those silos.
Role of the Product Manager
I think that product managers or product people really have two things that they are accountable for.
One aspect is your traditional textbook definition, “you’re the voice of your consumer.” You do a lot of product marketing. Some of the quotes are, “You’re the CEO of your product.” And I think that is absolutely important. You have to prioritize the backlog. You have to make sure you’re meeting the business objectives that your product actually entails.
But I think the other side of this that goes under the radar is that you need to spend just as much time ensuring that your development team knows what you’re building, why you’re building it, who you’re building it for so that they can build the right solutions.
Now, this becomes even harder when you aren’t necessarily building for your core customer.
At Target, I always think of our core customer as a suburban mom with a family. My products, GraphQL, Caching, and Data Services don’t service suburban mom with a family, they service engineers that are using my data or using my platform. While it becomes really hard to do that, but I think it’s so important to get grounded in those core products and say, “Here’s who our customers are. Here’s who our competition is. Here’s what they care about,” regardless of who they are.
Likewise, I think it’s really important that product people let your development team and engineers do what they’re really good at — problem-solving, and creativity. Let them own and drive how you’re going to achieve these objectives. Don’t get so caught up in the detailed features and functions. Let some of that go.
Step up, get onto the higher level of here’s what we’re building. Here’s why we’re building it. Here’s how I’m sequencing my things, and why I’m sequencing it this way. Have that healthy dialog of trust and empowerment to let your development teams build what they need to build.
Advice to Engineers with Skeptical PMs
Now, I’m not an engineer, but I would give this advice to engineers that have skeptical product counterparts.
Do whatever you can do to get grounded in the What, Why, and Who. Ask your product person or your end users, ‘why are we doing this?’ Really understand these components.
I really came out of my shell when I had a lead engineer say, “Hey, here’s what your objectives are. Here’s why we need to do this particular thing (pay down a debt, make switches, etc.) Because not doing so is sucking up our velocity and it’s going to stop us from being able to deliver what we need to next, and it’s that next thing which aligns to our vision.” If you have creative ways to articulate that why doing something, and how it will align to what the overall goals and your end users and benefit are, the better off you will be.
What does awesome product look like?
What it really looks like is that you have an empowered team. I think that’s the key here to get to awesome products, but really, some of the detailed things are the product team drives your backlog.
It’s not that your product owner writing every story in your backlog and tagging it in feature versus debt. It’s that your team all contributing, all saying, “Here’s what we’re trying to do in this iteration. Here is why this is important right now,” because of what your users have articulated to them.
Also, things not being labeled as debt, or stability or features, is also important. You might want to do that for reporting and measurement purposes, but this is just your backlog. It’s a list of stuff you guys need to do to enable the value that you’re driving.
The obstacles, that still remain for us.
When we started at Target on our DevOps journey our goal was to break down silos within our IT organization and begin driving the empowerment of engineers. I actually think those same challenges are still right here with us. It’s just more of that bigger, broader community.
It’s your product team and your business teams doing those things: breaking down the silos, empowering those teams to deliver the right things. And you’re never really done. Software is never done, it’s only ready to be shipped.