Part 4 of 4: Where to Start with DevOps Series
In this series, we have explored a variety of ways to think about your DevOps transformations
Week 1 we broke down the three key components to consider in choosing a starting place: Selecting Which Value Stream to Start With
Week 2 we covered how value is delivered to the customer and how to improve flow: Understand the Work in Our Value Stream and Improving Flow
Week 3 we learned about eveloping the habits and capabilities in people and the workforce as a means of facilitating these structures: How to Design with Conway’s Law in Mind
In the final post of this series, our goal will be to enable market-oriented outcomes where many small teams can quickly and independently deliver value to the customer.
When done correctly, Ops can significantly improve the productivity of Dev teams throughout the entire organization, as well as enable better collaboration and organizational outcomes.
To begin, let’s look at how shared services can create increased developer productivity.
One way to enable market-oriented outcomes is for Operations to create a set of centralized platforms and tooling services that any Dev team can use to become more productive – such as getting production-like environments, deployment pipelines, automated testing tools, and so forth.
By doing this, we enable Dev teams to spend more time building functionality for their customer, as opposed to obtaining all the infrastructure required to deliver and support that feature in production.
All the platforms and services we provide should (ideally) be automated and available on demand, without requiring a developer to open up a ticket or manually perform work. This also ensures that Operations doesn’t become a bottleneck for their customers.
By creating this effective internal marketplace of capabilities, we help ensure that the platforms and services we create are the easiest and most appealing choice available (the path of least resistance).
Creating and maintaining these platforms and tools is real product development—the customers of our platform aren’t our external customer but our internal Dev teams.
Like creating any great product, creating great platforms that everyone loves doesn’t happen by accident. An internal platform team with poor customer focus will likely create tools that everyone will hate and quickly abandon for other alternatives, whether for another internal platform team or an external vendor.
Often, these platform teams provide other services to help their customers learn their technology, migrate off of other technologies, and even provide coaching and consulting to help elevate the state of the practice inside the organization.
These shared services also facilitate standardization, which enable engineers to quickly become productive, even if they switch between teams. For instance, if every product team chooses a different toolchain, engineers may have to learn an entirely new set of technologies to do their work, putting the team goals ahead of the global goals.
In organizations where teams can only use approved tools, we can start by removing this requirement for a few teams, such as the transformation team, so that we can experiment and discover what capabilities make those teams more productive.
Internal shared services teams should continually look for internal toolchains that are widely being adopted in the organization, deciding which ones make sense to be supported centrally and made available to everyone. In general, taking something that’s already working somewhere and expanding its usage is far more likely to succeed than building these capabilities from scratch.
enablE product teams to become more self-sufficient by embedding Operations engineers within them.
These product teams may also be completely responsible for service delivery and service support.
By embedding Operations engineers into the Dev teams, their priorities are driven almost entirely by the goals of the product teams they are embedded in—as opposed to Ops focusing inwardly on solving their own problems.
As a result, Ops engineers become more closely connected to their internal and external customers.
Furthermore, the product teams often have the budget to fund the hiring of these Ops engineers, although interviewing and hiring decisions will likely still be done from the centralized Operations group, to ensure consistency and quality of staff.
For new large Development projects, we may initially embed Ops engineers into those teams. Their work may include helping decide what to build and how to build it, influencing the product architecture, helping influence internal and external technology choices, helping create new capabilities in our internal platforms, and maybe even generating new operational capabilities. After the product is released to production, embedded Ops engineers may help with the production responsibilities of the Dev team.
They will take part in all of the Dev team rituals, such as planning meetings, daily standups, and demonstrations where the team shows off new features and decides which ones to ship. As the need for Ops knowledge and capabilities decreases, Ops engineers may transition to different projects or engagements, following the general pattern that the composition within product teams changes throughout its life cycle.
This paradigm has another important advantage: pairing Dev and Ops engineers together is an extremely efficient way to cross-train operations knowledge and expertise into a service team. It can also have the powerful benefit of transforming operations knowledge into automated code that can be far more reliable and widely reused.
However, there may be a variety of reasons, such as cost and scarcity, where we may be unable to embed Ops engineers into every product team.
In these situations, we can get many of the same benefits by assigning a designated liaison for each product team.
Just like in the embedded Ops model, this liaison attends the team standups, integrating their needs into the Operations road map and performing any needed tasks.
We rely on these liaisons to escalate any resource contention or prioritization issue. By doing this, we identify any resource or time conflicts that should be evaluated and prioritized in the context of wider organizational goals.
If we find that Ops liaisons are stretched too thin, preventing the product teams from achieving their goals, then we will likely need to either reduce the number of teams each liaison supports or temporarily embed an Ops engineer into specific teams.
Once Ops engineers are embedded or assigned as liaisons into our product teams, our goal is to help Ops engineers and other non-developers better understand the existing Development culture.
To proactively integrate them into all aspects of planning and daily work, we can:
- Have Ops attend the daily standup
- Invite Ops to our Dev Retrospectives
- Make relevant Ops work visible on shared Kanban boards
As a result, Operations is better able to plan and radiate any needed knowledge into the product teams, influencing work long before it gets into production.
Let’s take a look at each of the rituals more closely…
Have Ops attend the daily standup
One of the Dev rituals popularized by Scrum is the daily standup, a quick meeting where everyone on the team gets together and presents to each other three things: what was done yesterday, what is going to be done today, and what is preventing you from getting your work done.
The purpose of this ceremony is to radiate information throughout the team and to understand the work that is being done and is going to be done.
By having team members present this information to each other, we learn about any tasks that are experiencing roadblocks and discover ways to help each other move our work toward completion. Furthermore, by having managers present, we can quickly resolve prioritization and resource conflicts.
A common problem is that this information is compartmentalized within the Development team. By having Ops engineers attend, Operations can gain an awareness of the Development team’s activities, enabling better planning and preparation.
For instance, if we discover that the product team is planning a big feature rollout in two weeks, we can ensure that the right people and resources are available to support the rollout.
By doing this, we create the conditions where Operations can help solve our current team problems or future problems before they turn into a crisis.
Invite Ops to our Dev Retrospectives
Another widespread ritual is the retrospective.
At the end of each development interval, the team discusses what was successful, what could be improved, and how to incorporate the successes and improvements in future iterations or projects.
The team comes up with ideas to make things better and reviews experiments from the previous iteration. This is one of the primary mechanisms where organizational learning and the development of countermeasures occurs, with resulting work implemented immediately or added to the team’s backlog.
Having Ops engineers attend our project team retrospectives means they can also benefit from any new learnings.
Furthermore, when there is a deployment or release in that interval, Operations should present the outcomes and any resulting learnings, creating feedback into the product team.
By doing this, we can improve how future work is planned and performed, improving our outcomes.
Feedback from Operations also helps our product teams better see and understand the downstream impact of decisions they make.
When there are negative outcomes, we can make the changes necessary to prevent them in the future. Operations feedback will also likely identify more problems and defects that should be fixed—it may even uncover larger architectural issues that need to be addressed.
Make relevant Ops work visible on shared Kanban boards
Often, Development teams will make their work visible on a project board or kanban board. It’s far less common, however, for work boards to show the relevant Operations work that must be performed in order for the application to run successfully in production, where customer value is actually created.
As a result, we are not aware of necessary Operations work until it becomes an urgent crisis, jeopardizing deadlines or creating a production outage.
Because Operations is part of the product value stream, we should put the Operations work that is relevant to product delivery on the shared kanban board.
This enables us to more clearly see all the work required to move our code into production, as well as keep track of all Operations work required to support the product.
Furthermore, it allows us to see where Ops work is blocked and where work needs escalation to highlight areas where we may need improvement. When we do this well, we achieve market-oriented outcomes, regardless of how we’ve drawn our organization charts.