InnerSourcing Program: Scaling Automated Governance Chapter 2

“That’s great, Mira! And you, too, Priya. That kind of open-mindedness and willingness to work together is exactly what we all need to model.”

Everyone’s heads turned to look at the woman seated at the end of the table closest to the door.

“Ah, Jane! I’m so glad you could make it,” Jordan said, as he clasped his hands together in excitement. “Everyone, I invited Jane from BAD to join our retrospective today because I think she can help us get on the same page.” 

Jordan had met Jane at a conference several years before the merger of BAD and FIN. She was impressive. In one moment, she was stumping those around her with complex technical proposals and questions, and in the next, she would be inspiring the group with novel insights, enthusiasm, and energy. When he found out that she worked at BAD, he knew they would have a kindred spirit to help implement and champion the changes they needed.

“Thanks, Jordan!” Jane nodded, acknowledging the introduction. “We’re all working toward the same goal, but up to this point, FIN and BAD have still been operating as independent organizations. As friction between our working styles and corporate cultures has been continuously ignored in favor of focusing on delivery, any opportunity to build a collective trust relationship has been completely eroded. We need to start from scratch, and we need to start with people, not technology and process.”

Jane paused to let everyone take this in. After no one seemed poised to interject, she continued. “First and foremost, can we all agree that everyone here wants to see our company succeed and that we’re all acting with the best of intentions given the resources and knowledge available to each of us?” 

This was met with more silence, but after a few hesitant moments, heads around the room started to nod up and down.

“Wonderful. Now, with that in mind, let’s try to tackle our technical and procedural challenges as a unified team. Instead of FIN’s way versus BAD’s way, let’s be FIN and BAD versus the problem at hand. And I have an idea that we can use to help with bringing us together better. We have been working on this InnerSource@BAD project within the business development unit. The project was started to allow us all to share a set of our capabilities around how we are using machine learning to identify investment opportunities and grow our client base. We discovered that many different groups were using ML in various ways and many of those teams were rebuilding what already existed. We knew if we were going to make use of ML effectively, we would need to bring teams together, but we couldn’t just pick one solution, as there were many different use cases. This is when we started thinking about how open-source works. 

Open source refers to a type of software or technology that is developed collaboratively and made available to the public, allowing anyone to view, modify, and distribute the source code. The key characteristic of open source is its transparency and accessibility to the wider community.

Jane continued. “We decided to bring the open-source idea inside the company. We established an InnerSourcing program that all these groups started using. It’s been amazing. We went from forty totally different ML and AI project areas to only ten unique innersource projects. Maybe we can apply the same concepts to help us here?”

Jordan was already taking notes on Jane’s suggestion, excitedly typing away on his laptop while watching and nodding attentively. “Wow,” he said, briefly pausing his typing, “that must have taken a lot to get people working together. What did you do to make it happen?” 

Jane continued her pitch. “We started by creating a project in our git server internally that could be visible to any developer in the company. Then a few of the teams that were already pretty common and had a large business need for new capabilities came together to manage their code in this common repo. It took some work to get the group set on the main contributors and collaborators. Each of the participating teams had to see the value to their business application and had to see they were being listened to in capabilities needed. But with these first few teams, we got it started.”

Mira was still a bit skeptical of the approach. “But how did you get them to collaborate together?” she asked.

“Honestly,” Jane explained, “we got a few of the teams that were most open to change and those that had the largest commonality. We could explain the common goal: better solutions for all and an easier way to get the function they needed because they could spread the work. Basically, we explained to each of them that they were growing their overall team with the other resources. I know that’s not what was really happening, but it got each team willing to work toward the goal. And by doing this they were able to work together.”

Jane paused and smiled. She closed her eyes for a moment as if to summon the past experience. “I think the key was that we started small. It allowed us to get a win so the teams would start sharing how useful it was. Word started spreading and pretty soon other teams started to join in. This is when we ran into some problems with this approach. We needed some level of guardrails to keep the program moving forward in the right direction so that this growing list of products that depend upon this InnerSourced code would continue to work.”

Actionable Guidance

When creating an InnerSource program, start small with highly interested teams.  

Have guardrails for contribution and use. 

“Wow!” Mira liked what she was hearing. The thought of this level of technical collaboration across BAD seemed foreign but absolutely delightful. Maybe there was hope here after all! “That’s incredible, Jane!” She thought for a moment and wondered if they could have done the same thing with their automated governance project when they introduced it at FIN. “What did the program do to ensure the code quality was high enough not to break any of the dependent products?”

Jane laughed! “Well, I confess, we didn’t get it right at first! In fact, it took quite a few iterations before we reached the level of quality everyone was happy with.” She continued, “We took the lessons learned and created a simple set of initial contribution guidelines. Every InnerSourced product has a dedicated set of maintainers whose day job is to keep the product updated and secure. They also review all contributions. Anyone can contribute. Contributions include automated tests and enough documentation for the new function or the fix to make it clear.”

Cora joined in. “The idea of having a core set of maintainers has been true for a number of projects I have worked on as well. However, as one of those maintainers, I always get contributions that don’t even pass the basic tests. How did you enforce the automated tests and other guidelines?” 

“That’s actually what created our second major InnerSource product,” Jane said, smiling. She was delighted to see the mood swing in the room. “We needed a standard automated pipeline that could enforce the security standards within the company as well as the additional standards we wanted for InnerSource. We had started with one team’s pipeline, but other teams wanted to use different tools. We decided to create an InnerSource pipeline product so people could contribute to the various tasks that were needed.” 

Bryson had been listening for a while and decided to chime in. “Just creating an InnerSource version of the pipeline doesn’t solve the arguments about which tools to use. That just makes it even more complex!“

“Yes, you’re right.” Jane turned to face Bryson. “We can’t actually solve that problem as part of InnerSource. That’s a larger issue to deal with as part of the standardization of the tools. But for the InnerSource projects, we defined a configuration-based definition for the pipeline within the application repositories that lists the set of tasks that need to be run as part of the pipeline.” Bryson had a confused look on his face so Jane tried to clarify. “The configuration specifies what needs to be done but not in a tool-specific way. All the task automation is created in a tool-neutral way. This way people don’t argue over the pipeline tool and instead focus on what needs to be done.” 

Actionable Guidance

Pipeline is a good InnerSource project and provides automated guardrails. 

Use configuration for automation steps to facilitate technology changes.

Bryson was impressed. Jane was still looking at him with a huge smile on her face. We sure needed someone like Jane months ago, Bryson thought as he looked over at Jordan and grinned. It was obvious that he was just as enthused. 

Jordan nearly shouted, “This is great, Jane! How can InnerSource help our current solution? You talked about AI/ML and pipeline automation tasks, but that’s just a few areas. Do you think we can use this approach to scale the FIN automated governance to all of BAD?” 

“Actually, I do,” Jane replied. “The AI/ML pipeline was a big start. It made it clear that we needed a plan for more InnerSource products and for a way of discovery. We knew we would need a way to easily find products that already existed and a way to contribute new products. But there had to be some level of governance to make sure people reviewed what already existed before considering submitting something entirely new. We needed commitment that the new products would have a core set of maintainers to keep them secure and updated. New products also had to be something that would be shareable. It had to be a function that other teams would need. And our pipeline product capabilities were required for all submissions. It included all the right security scanning, build process, and checks required to demonstrate the products were usable by teams without a ton of extra overhead. Bringing in the pipeline with the code security scanning and SBOM creation was the best next step we could have ever made.”

Intrigued, Bryson asked, “But how did that help with visibility and discoverability?”  

Jane answered, “The pipeline had a developer portal. That’s what provided the foundation for our InnerSource portal. We just had to add additional capabilities for product discovery,  onboarding, and contributions. All new products could then use that as a starting point.”

Doug had been listening intently to the conversation. He was aware of Jane’s efforts but wasn’t yet convinced it would help solve the problem of scaling automated governance. “This is great Jane, but in my experience, one of the biggest problems with using open-source software is ongoing support. You decide to use it then realize it is out of date and is no longer supported. Don’t we have the same problem with InnerSource?”

 “Yes, great point, Doug.” Jane smiled, happy to see Doug join in. “And, yes, that was a major problem. We had some very successful products and some not so successful. Those unsuccessful efforts led to learning and change. We developed a set of guardrails to help keep things on track as well as a small dedicated team to make the InnerSource program successful.”

Doug thought for a minute, then asked, “So how do we control what products get added to the Innersource program?” 

“We added a gate!” Jane almost laughed, knowing it would get a reaction from the FIN team. “But just enough of a gate, not a heavy blockade. We still want to encourage all teams to bring their candidate products to the InnerSourcing program.” Some skeptical looks from the team prompted her to continue. 

“We tried having teams do their own checking, but we soon discovered everyone thinks their own solution is unique and were not really bothering to look at other products that may work for them. We introduced a simple, relatively painless step of requiring teams to submit an issue to describe their product and begin the onboarding. This kicked off a process where the InnerSource team checked this new product request against existing ones. If no overlap was discovered, the request was approved and the team was notified. They were then able to kick off the automation to move their product into the standard format. Plus, it automatically updated the documentation. The InnerSource team would make recommendations about documentation improvements that should be made to help their product gain more interest and contributors from the company.”

Bryson loved this. It was clear that the gears were turning in his head. “I like this approach, but just like any open-source project, do you have problems with the product teams abandoning their efforts after they get it working?”

“Absolutely,” Jane answered. “It’s a real problem. Keeping the product team engaged to maintain it can be a challenge, especially for those useful hackathon projects that become outrageously popular. To address that, we put a policy in place to remove any product from the InnerSourcing program that does not stay current. In other words, a product that uses a vulnerable dependency, perhaps an open-source library, or has a growing list of unaddressed security issues will be removed. The first project we ended up removing was actually used by a number of business applications. They yelled quickly, and then we were able to make it clear there is no such thing as totally free. They each approved some level of maintainer support and the project was restored. To be fair, we don’t actually remove anything. We just hide them and remove them from the menu because as we all know people will come out of the woodwork saying they are using something.”

Bryson said, “Okay, so far so good, but tell me this. How do people know what products are available? How do they see what exists?” 

Jane smiled, “Ah yes, that’s where our portal comes in…Which, by the way, is also an InnerSource product. The portal lists all of the available products. It’s our ‘InnerSourcing menu’ and a big feast of awesome!” Jane laughed and everyone joined in. The mood was improving and the FIN team was encouraged. 

“You’re going to love this.” Jane continued, “The portal includes contribution guidelines, ways to consume the products, and how to create new products. It also outlines the process for how products are removed from the InnerSource portal. But more importantly, it has automation to help convert existing projects into the standard format. This includes documentation templates. Teams have actually started using this as their standard template even when not submitting for InnerSource because it makes it easier to turn their product into an InnerSource contribution later.”

Bryson had another thought. “That sounds helpful, but do you get good-quality contributions?” 

Jane answered, “Great question. It takes time and some planning upfront. We created a set of contribution guidelines to help contributors understand the process. When it first started, we had people doing pull requests and expecting their code to be integrated right away, without comments or feedback. We had to make it clear that contributions must include tests to validate the contribution worked, and they must have a clean pipeline run to show they did not break existing functions. There are situations where the old tests can be the problem, and those tests need updating as part of the pull request. Documentation updates are another critical part. If a significant new function is being added, it needs to be explained.”

Bryson started thinking about the marketing effort to get all of FIN and BAD teams engaged with this program. “How do people even know about this InnerSource program?” 

“Well, that, for one!” Jane pointed to an “InnerSourcing@BAD” poster that was across the room. It was slick. Someone from the marketing team had put together an eye-catching, high-tech poster that looked more like a movie poster than an enlistment reminder. She laughed and then added, “Another important step was to start an InnerSourcing guild. This team helped spread the word and encouraged teams to participate. It resulted in more contributions and adoption of products in the program. For one, it helped promote the common CI/CD pipeline across the various technology teams. We started seeing contributions coming for all areas.”

After a moment of thought, Cora piped up, “Hey, wait everyone, I’m excited about this InnerSource idea, too, but is it the silver bullet that will solve all of our problems? It will definitely help address many of our concerns, but I think there’s more for us to discuss. One of the big complaints I heard from BAD teams about the rollout of the FIN automated governance solution was that it was incompatible with their tools. Adopting this InnerSourcing approach for our solution will help with adoptions and encourage customization, but if our developers and business teams are all using different tools, won’t we continue to hit roadblocks?”

Jane responded. “I agree, I think we still need to solve the tools problem. It is definitely an area that is preventing great collaboration”

Bryson agreed. “Our AutoGov system was written to be flexible but not this flexible. The variety of code and artifact repositories, pipeline orchestration software, and ticketing systems from multiple generations is making integrating it here impossible. You may be asking yourself, why do we have so many varieties and generations of tools?”

“Well, good question.” Bryson answered himself and grinned, “There’s a whole mess of reasons that we find ourselves in this state. We’ve given teams the freedom to pick their own tools and they’ve done exactly that. This is what happens. We have legacy tools that have not been decommissioned and we have new tools introduced from some of our smaller acquisitions. It’s not just pipeline software either. We now have, within FIN, all the tools and software that have been collected over the years as well as the new additions from merging with BAD.”

Bryson looked around the room and could see that the team was exhausted. “It’s late,” he said. “I think we made some good progress today talking about how the InnerSource program that BAD has introduced can extend to all we do as a new, combined company. I suggest we table the discussion on tools and regroup tomorrow to figure out how to manage the ‘tool zoo.’” 

“I second that,” said Jane. Others joined in agreement. Unlike at the beginning of the day, the team seemed to be in a better mood. Mira was even smiling. Perhaps there was some hope after all.

Stay tuned for the next exciting chapter…

- About The Authors

Jason Cox

Jason Cox is a champion of DevOps practices, promoting new technologies and better ways of working. His goal is to help businsses and organizations deliver more value, inspiration and experiences to our diverse human family across the globe better, faster, safer, and happier. He currently leads SRE teams at Disney and is the coauthor of the book Investments Unlimited. He resides in Los Angeles with his wife and their children.

Follow Jason on Social Media

• 
• 

Sean D. Mack

Christina Yakomin

Senior Cloud Architect at Vanguard

Brian Scott

Brian Scott is a seasoned technologist with over 25 years of experience in DevOps, SRE, and managing technical operations at scale in Cloud & Infrastructure. His career includes impactful roles at MySpace, OpenTable, and The Walt Disney Company. Currently, as a Principal Architect at Adobe, Brian supports engineering teams with technology, cloud, and AI governance and adoption while assisting senior leadership in solving enterprise-wide challenges and breaking down technical barriers.

John Willis

John Willis has worked in the IT management industry for more than 35 years and is a prolific author, including "Deming's Journey to Profound Knowledge" and "The DevOps Handbook." He is researching DevOps, DevSecOps, IT risk, modern governance, and audit compliance. Previously he was an Evangelist at Docker Inc., VP of Solutions for Socketplane (sold to Docker) and Enstratius (sold to Dell), and VP of Training & Services at Opscode where he formalized the training, evangelism, and professional services functions at the firm. Willis also founded Gulf Breeze Software, an award winning IBM business partner, which specializes in deploying Tivoli technology for the enterprise. Willis has authored six IBM Redbooks for IBM on enterprise systems management and was the founder and chief architect at Chain Bridge Systems.

Follow John on Social Media

• 
•