Taming the Tools Zoo: Scaling Automated Governance Chapter 3

“Look, I understand what you’re trying to do here,” Priya interjected. 

The team had reassembled to recap their progress and discuss the other hurdles they needed to clear for FIN and BAD teams to start working together. Jordan had congratulated the team on the progress and applauded Jane’s pitch to expand the InnerSourcing program across all of FIN. Bryson had just started outlining the concerns about the “tool zoo” that they needed to manage. His proposal was to restrict and standardize a slate of tools for the combined company to use. 

Priya continued, “We need to carefully understand just how far we should take this. We have been trying to get a new marketing design tool for the past six months. This tool would allow our product leaders and designers around the world to collaborate in real time. And it’s not even a big expense! We’re talking maybe $20k a year.”

“But that’s something different,” interrupted Doug. “That’s a bureaucracy issue. I’ve been talking about bureaucracy busting for years but nothing seems to change.”

“Well, that’s part of it,” Priya continued unphased. “But that’s not all of it. After six months of getting bounced back and forth between legal and procurement, I finally got directed to someone in sourcing, and they told me that I couldn’t purchase it because it wasn’t one of our ‘standards.’ The funny thing is, when I asked what the standard was, I was told that as this was a developing area the standard hadn’t been set yet.” Priya had recently joined BAD before the merger. Her prior role was leading the marketing team at a FinTech startup. She had welcomed the opportunity to apply her skills and ambition at BAD but had soon been hit with the reality of a large and debilitating corporate culture. The acquisition by FIN had given her new hope, but here they were, talking about global corporate standards that could suffocate innovation.

“I’m used to being able to make these decisions,” she continued. “In the past, if I needed a marketing tool for my business, I went out and bought one. I mean, I did my due diligence to be sure, but it was in a matter of weeks, not months on end. I understand the need for standardization, but what you’re suggesting—this over-standardization—this has the real capability to kill innovation. We need to be able to quickly make localized decisions about technology that is right for our business.” There were murmurs of agreement and disagreement throughout the room. Priya had clearly touched on a nerve.

“I think you’re onto something there.” From the back of the room, Jane’s voice cut through the noise. “We need to allow for autonomous decision-making, but we can’t do that if it is detrimental to the benefit of sharing the same toolset. And we certainly can’t block innovation and experimentation when we don’t even know what the right solution is.” Jane paused to see if everyone was following. “Remember, we were told not to crush the butterfly. We can’t put governance in place that is going to stifle the reasons why we merged with FIN to begin with.”

“I think we need to refer back to the idea of paved roads and bridges.”  Jane walked from the back of the room to the front. The group was sitting around the conference room table facing a dry-erase board and monitor. Priya looked on intently as Jane spoke. Jane wrote “Paved Roads and Bridges” on the board. “We need to build the roads and bridges that enable our technology teams to get where they’re going quicker. Specifically, these paved roads and bridges provide engineers with a smooth ride to delivery while automatically meeting security and infrastructure requirements. We want to make it easy to do the right thing” 

Mira thought about it for a moment. Something was puzzling her. “But what if that road doesn’t get us where we want to go?” she asked, trying to understand how this fit the current solution. 

“Then we should allow them to go off-roading or build their own roads. That is exactly the point!” Jane raised her arms to drive home the point. “And that is how innovation should occur.” 

“I like this idea,” Bryson interjected, “but I think folks need to understand that, if we allow them to go ‘off-roading,’ they’re on their own because our tow trucks can’t reach them off the beaten path.”

Jane continued. “Yes, and if enough people follow that road, then we should codify it by paving that road and making it accessible for everyone. In that way, we encourage people to use the same path and reduce costs through re-use, all while allowing for experimentation and innovation.”

Everyone sat for a minute absorbing the concept. Janelle scratched her chin absorbing the information but there was something bothering her. “This is great guidance, Jane, and it provides a very useful starting point but what are we going to do about the problem? We have a problem today and this concept doesn’t fix that.” Jordan had invited Janelle to join the discussion. He wanted her input as the Chief Risk Officer (CRO) for the combined company. Janelle and her team had been instrumental in helping launch automated governance at FIN.

This was bothering Mira too, but something occurred to her. “You know, I tend to agree. We need to take action in the direction Jane has set. But that gives me an idea. I was recently reading a book called Good Strategy/Bad Strategy by Richard Rumelt. In his book, he says that a good strategy has three elements: diagnosis, guiding policy, and coherent action.”  

Mira joined Jane at the front of the room and picked up a marker. “Now, we know the problem. We’ve discussed that again and again. And, it seems like Jane has given us the ‘guiding policy,’ this ‘paved roads and bridges approach.’ But what we don’t have are coherent actions to tie the two together. What are we going to do to address this problem?”

Actionable Guidance

Define a guiding policy that will drive your tools management process.

Make it easier to do the right thing by building “paved roads and bridges.”

“Wait,” Bryson interjected. “Before we all go skipping down the yellow brick road, I’m not sure we really know the problem. It’s not just getting things out the door, it’s keeping them available. There’s a run-time issue here too. I just spent the last two hours and I don’t know how many brain cells trying to help resolve a major incident with our online investment portfolio. We tried to help but we had no visibility into the problem. We don’t even have access to the monitoring systems they’re using,” Bryson was still visibly frustrated. 

Doug nodded in fervent agreement. “Not only that, we couldn’t even speak to the customer services team to find out what the problem was.”

“What do you mean the developers couldn’t communicate with the customer services team?” Janelle asked.

Doug took a deep breath and explained. “Well, our developers are on Smack Messenger and the Customer Services team is using ChatSquad.”

Janelle sighed and shook her head. “We need to get everyone on the same systems! We need to pick one and stick with it. For the WHOLE company. Not one chat or email system for you and another for you. We need to standardize across the board. We can’t behave as one company if everyone is using a different set of tools.” 

“That sounds great on the surface, but it isn’t as easy as you think. The software build system is integrated with Smack, including developer notification, approval processes, and automation bots. On the flip side, from what I hear, the Customer Service team has no training to use Smack. I think we will find the same challenges with other tools too.” Doug shook his head.

“It sounds like it isn’t going to be easy, but shouldn’t we have senior leadership just dictate that everyone must use one tool?  Perhaps that should even extend to other areas of technology sprawl . . . ” Janelle thought for a minute and laughed, “one public cloud, one operating system, one development language, one integrated development environment, one ring to rule them all?” 

“Wait, you mean you want me to get all my developers all using the same IDE?!” Now it was Doug’s time to be upset. He had seen this before. An effort to over-standardize so blatant that it could kill productivity and disempower his best engineers. “If this makes it harder for our engineers to do their jobs, we’re going to have a revolt. We’re having a hard enough time keeping great engineers as it is. Look, if we don’t trust them enough to choose their own IDE, why did we hire them in the first place? And isn’t DevOps all about trust and empowerment? This is like hiring a Michelin-star chef and then telling them they need to use the same knives that everyone else uses because those are the lowest priced and most durable ones the restaurant can buy.”

There was dead silence in the room for a minute. Mira was surprised to find herself actually agreeing with Doug for once. “I agree with Doug. That’s a great example of where independent choice is more important, a choice that doesn’t affect others. On the other hand, if everyone wants to use a different type of oven, you would have hundreds of ovens and no room for the cooks. So maybe there are places where independence makes sense and places where it doesn’t.”

“That’s exactly right, there is no one-size fits all solution here,” Cora agreed. “The right level of variance in our toolset is dependent on a whole host of considerations.” 

“Everyone, hold up!” Doug blurted out. “We all are describing a seven-headed monster. We need a process that is less bureaucratic, allows for localized decision-making, enables innovation, but at the same time enforces standardization, reduces risk, and jives with our procurement processes?!”

“Exactly!” exclaimed Mira. Everyone in the room laughed except for Doug, who obviously wasn’t sold that this was even feasible.

“I think this is exactly what we need, and I don’t think it’s impossible. But it’s an elephant, and we need to take it one bite at a time,” said Mira.  

“Stop,” said Jane rolling her eyes. “It’s almost lunch and I’m hungry.” There were more laughs heard from around the room, which was a good sign that folks were approaching this topic from a positive perspective.

Priya stood and walked over to one of the flip charts and untopped a marker. “Let’s break this down. What do we need?”

Janelle began, “Let’s start by identifying the key areas of our business that require different software technologies.”

Actionable Guidance

Assess problem space and the amount of tool sprawl.

“Nice,” said Priya, scribbling down this point. “What else?”

Jane added, “We should be able to categorize our software technology into high and low risk areas. This will help us to identify areas where we shouldn’t allow for as much flexibility and areas where we don’t need as much conformance.”

“OK, but I’m a little unclear by what you mean by risk in this context,” said Doug.

“Well, I guess I mean risk to the business. More established portions of the business, like our loan division, perhaps, which have an established customer base and could suffer dramatically if they have major changes are pretty risk intolerant. Whereas new, emerging segments of the businesses like our AI financial advisory service, which we just launched, are more risk tolerant because they have fewer clients and no financial transactions. In fact, I’d say areas of the business like this need to take risky moves in order to succeed in a new market. We need to enable these areas to be flexible with their toolsets.”

Janelle responded, “We could designate certain categories as innovation areas where we should be experimenting with new technologies and testing their benefits and match. But we still need to ensure that we’re not taking any unnecessary risks.”

Doug interjected, “OK, I like where this is going. But how do we determine which areas are high risk and which are low risk and can allow the freedom to experiment with new technologies?”

Janelle responded, “How about we develop a risk framework with criteria to assess risk based on factors, such as the potential impact on our customers, data security, regulatory compliance . . . things like that.”

Bryson nodded, “That’s good, but I would want to see some non-functional types of criteria, like scalability, resiliency, and operability. We need to ensure that the software technology we select can support our business operations in the long term.”

Doug added, “What about the cost? We need to consider cases where we have similar tools that should be rationalized. By combining similar products, we can negotiate the best cost for a product based on a larger enterprise agreement.”

Priya nodded and wrote down the point.

Jane said, “I like this. Feels like this could strike a balance between risk management and innovation.”

Actionable Guidance

Assess the risk of the business area, risk of the tool, and the business life cycle stage (emergent vs. mature) to guide tool selection.

Doug added, “And, I think we need to consider network effect.”

“Network what now?” Bryson asked.

“Network effect. You know, the idea that the more people who use a product the more its value increases,” Doug explained. “So, for example with chat, there’s a high network effect because the more people who use the same system the more easily information will flow across the organization. IDE’s, on the other hand, have a pretty low network effect. If I choose a different IDE than you, it should have limited impact on how you work or how we work together.” 

“That makes sense,” Bryson responded. “But we also need to think about the size of the team and divisional autonomy. And there’s a pretty fundamental question here as to whether we are acting as one completely integrated company or as a federated collection of companies.” 

Bryson could tell by the reaction in the room that some didn’t agree or may have misunderstood him. He clarified, “Look, I know that we are trying to integrate BAD. We clearly see that there are benefits of working together and that we are stronger if we are one company. That’s why we made the acquisition. But that may not always be the case. There may be acquisitions that are focused on customer acquisition and gaining a larger share of the market where technology integration may be less important.”

“That actually makes a lot of sense,” Priya agreed. “So, we need to think about risk, cost, product maturity, company structure, team size. Anything else?”

“No, I think that’s a good list.” Janelle thought for a minute. “But I still think we need to be careful. Some people seem to jump to new technology just because it’s a shiny new thing or because they used it in their last job, even if it doesn’t offer any appreciable difference from tools we already have.” 

“Right! And that is exactly why we need a process to help manage this sprawl, to look at new options and assesses against our current capabilities.” Jane was excited to see the team engaged and contributing to the discussion.

Actionable Guidance

Determine which tools need to be strictly standardized and which can have variability. The right level of variability in tool selection will be based on considerations such as scale, maturity, network effects, and company structure.

Jane continued, “When considering new or existing software, we just look into the category to see what else already exists and whether there is a need for new software or a need for factor-
based evaluation. We could even assign an owner to each software category who would be responsible for deciding whether an evaluation is necessary and, if so, would be responsible for approving new software. It’s like a paved road inspector.”

Mira smirked, “Well I’m going to want a big, shiny badge if I’m nominated for the job.”  Everyone chuckled.

Doug nodded, looking into the distance as if considering everything he had heard. “So, by creating this architecture model, an evaluation methodology that covers our range of needs and risk levels and ownership responsibilities, we can ensure that we’re selecting and keeping the right software for our business while also managing risks effectively. It will, at the same time, allow us to strike a balance between innovation and risk management while also contemplating the network effect impacts of our decisions.”

Everyone in the room nodded. Jordan smiled at Jane. They were making progress!

Actionable Guidance

Develop standards reference model.

○ Include recommended standards for each category.

○ Include a process for introduction of new software to be evaluated against current standards. 

Leverage category teams who are accountable for a specific type of tooling.

Scratching his chin and squinting his eyes, Bryson agreed. “I think this is really good. But isn’t the problem that we’re not sure what software we have today between BAD and FIN? I mean, it seems like we are already at a point where we need to make some decisions.”

The good feelings and smiles that had developed over the last thirty minutes seemed to evaporate.

After a few awkward moments, Mira began, “It’s hard to argue with that, Bryson. We need this model pretty quickly, and then we need to figure out what software we already have.”

After some back and forth on all the ways that BAD could find out what software was being used, including looking at the obviously out-of-date software catalog, network sniffing, end-point agents, surveys, and looking through procurement purchase orders, the team felt they had solved the software discovery issue.

“OK, so now we have a taxonomy. We have a way of discovering what software we have, categorizing that software into the taxonomy, and evaluating it for all the dimensions we previously talked about. Finally we can decide which ones require decisioning and which ones don’t require attention. What now?” Priya asked.

“Great question. We built the seven-headed monster, but who’s going to feed it?” asked Doug.

“I told you I was hungry!” Jane said and everyone laughed. “But it’s a good question. We need to make sure we have an ongoing process for ensuring that new software purchases follow this same discovery model, where they are assessed for all of the dimensions we talked about and whether they require intervention.” 

“We also need to ensure we review the software we already have, so we don’t keep paying for outdated solutions just because we’ve always done it that way,” Bryson said.

“Yes!” Jordan agreed. “Not only do we need a way to review new tools that might enhance our capabilities, we also need to revisit old assumptions to make sure they still hold true.”

Amelia, head of risk and audit, who had been sitting quietly, piped up, “We actually had a process that solved this issue at my last company. We got all the software owners into a room once a month and went through all our applications to see what we still needed and what we could get rid of.”

Doug groaned. “Great! Another TEP-LARB.” 

“What’s a TEP-LARB?” Amelia asked.

Doug seemed annoyed at having to explain himself but also eager to describe the problem. “The TEP-LARB was a type of review board in the book The Unicorn Project, one of the foundational books of DevOps. I think it stood for Technology Evaluation Process something something Review Board or something like that. It was basically an architecture review board whose main purpose was to say no. It was a huge blocker to innovation in the organization,” Doug explained. “Sounds like that’s exactly what you’re proposing. I mean, aren’t we supposed to be empowering people to make their own decisions?”

“But it doesn’t have to be like that,” Mira interjected. “I mean, I hear you Doug, and I agree with the concern, but if we set this up as an opportunity to explore new solutions, it could be a place to drive innovation instead of stifle it.” 

“I, for one, like the idea. I think this might be our solution here,” Jordan interjected.

“In addition, eliminating old tools that aren’t doing what we need anymore should be something that everyone can get behind,” Mira continued. “And if we ensure that the people responsible for a category are there at the meeting representing what is best for that category and the company, then maybe it can actually be an empowering force.”

“OK, OK,” Doug conceded. “As long as we are setting it up to actually help the organization get what they need then I guess I can see how this could be a positive way to get our tool landscape aligned and under control.”

Actionable Guidance

Implement a technology selection committee to manage standards reference model.

Develop software/application life cycle and ensure that standards have an expiration date at which they need to be reviewed. 

Ensure that the life cycle management process includes the removal of items to continually prune old items to reduce software sprawl. 

“Wow, I think we may actually have something here,” Priya observed. “I guess what it comes down to is that there’s no ‘one size fits all.’ We can’t have everyone on one standard, but we also can’t have everyone choose their own. There’s a whole bunch of factors that play into whether we have one tool for a given area, five tools, or just let people choose what works best for them.” 

“But if we put these criteria in place, and have this process for helping align those decisions for the organization as whole, then we might just find the right balance,” Bryson observed.

Mira agreed, “And it’s just like Jane said, if we can make the standard tools the most effective and easiest, then people will naturally gravitate toward those anyway.”

Priya was happy. This had gone much better than she had thought. “Yeah, and if we review regularly, we can make sure we don’t get stuck with decisions that no longer make sense for the current environment.” 

Jordan summed up, “Team, this is amazing. Do you see what has happened? Over the past two days, we have tackled two seemingly immovable obstacles and made a way forward. We are already building trust and community with our rapidly expanding InnerSource program. And today, we now have a solid beginning to tame our unruly zoo of tools. Great job everyone!”

“I don’t know about everyone else, but I’m happy with our progress!” Bryson added.

“Well, if Bryson is satisfied, then we must be doing something right,” Mira joked. 

“I guess so. Now let’s get that bite to eat we’ve been talking about!” Bryson laughed.

Stay tuned for the exciting concluding chapter.

- About The Authors

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Sean D. Mack

Christina Yakomin

Senior Cloud Architect at Vanguard

Brian Scott

Brian Scott is a seasoned technologist with over 25 years of experience in DevOps, SRE, and managing technical operations at scale in Cloud & Infrastructure. His career includes impactful roles at MySpace, OpenTable, and The Walt Disney Company. Currently, as a Principal Architect at Adobe, Brian supports engineering teams with technology, cloud, and AI governance and adoption while assisting senior leadership in solving enterprise-wide challenges and breaking down technical barriers.

John Willis

John Willis has worked in the IT management industry for more than 35 years and is a prolific author, including "Deming's Journey to Profound Knowledge" and "The DevOps Handbook." He is researching DevOps, DevSecOps, IT risk, modern governance, and audit compliance. Previously he was an Evangelist at Docker Inc., VP of Solutions for Socketplane (sold to Docker) and Enstratius (sold to Dell), and VP of Training & Services at Opscode where he formalized the training, evangelism, and professional services functions at the firm. Willis also founded Gulf Breeze Software, an award winning IBM business partner, which specializes in deploying Tivoli technology for the enterprise. Willis has authored six IBM Redbooks for IBM on enterprise systems management and was the founder and chief architect at Chain Bridge Systems.

Follow John on Social Media

• 
•