DevOps Blog

Get the RSS Feed

The Three Ways: The Principles Underpinning DevOps

In this blog post, I talk about the “Three Ways,” which are the principles that all of the DevOps patterns can be derived from, which we’re using in both the  “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.” We assert that the Three Ways describe the values and philosophies that frame the processes, procedures, practices of DevOps, as well as the prescriptive steps.

It has been especially fun working on this with fellow co-author Mike Orzen, the most recent winner of the cherished Shingo Prize for his contribution to operational excellence for his book “Lean IT.”

The Three Ways are as follows.  In a future blog post, I’ll describe the DevOps patterns that can be derived from each.

The First Way emphasizes the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, system administrator).

The focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.

The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system (as per Deming).

The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.

The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.

The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.

We need both of these equally. Experimentation and taking risks are what ensures that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.

The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.

  • Pingback: Elements Of The First Way: And The DevOps Implications… | IT Revolution

  • DM Colburn

    I applaud Gene Kim for his contribution to the DevOps transformation effort. These 3 principles offer a valuable focus for evolving the culture change requirements that are the core challenge for embedding a DevOps value system. I do have a suggestion for discussion that may serve to enhance the value of these 3 principles.

    The focus on “Systems Thinking” is a concern, if we all agree that the biggest challenge to DevOps is changing current culture, values and rewards thinking. Wouldn’t the focus on developing and managing “systems” support an internal IT-centric perspective? Wouldn’t a focus on Dev and Ops co-designing “Services” better support a business-focused perspective on the value of IT? I submit that integrating the core values of DevOps into the IT Service Lifecycle would most certainly be better served by a ‘service’ focus for the 3 principles.

    For internal IT departments to remain relevant there must be a change in how the business values IT. IT must evolve from a ‘cost center’ or ‘spend-only’ department, to a ‘service provider’ or ‘contributing’ department. To be successful IT must be capable of demonstrating value in the same way, and at the same valuation points business departments use to demonstrate their value (i.e. optimizing business process.)

    Business department leaders are inherently familiar with and experienced in quantifying (valuing) ‘services’. ‘Systems’ on the other hand, are more
    commonly recognized as the purview of IT or manufacturing-related departments. Manufacturing-related departments however, have evolved a more mature capability to quantify their value in support of core business process. IT departments are struggling to evolve this same capability.

    DevOps is a critical paradigm shift required for IT to evolve the capability to value its services in a way business department leaders understand. Building serviceability, availability, reliability and service level quantifiability into the business process automation services IT designs, transitions, operates and continually improves, requires integrating development and operations as a single discipline functioning in all phases of the Service Lifecycle.

    For this reason I wonder if it would be more ‘IT Transformation’ supportive to include the concept of DevOps co-developing services as a core principal, avoiding the possibility the ‘systems’ perspective in the First Way above, would be seen as perpetuating the internal IT focus for DevOps, as opposed to a business-focused ‘service’ perspective.

    My intent is to facilitate discussions on this point, to enhance, not be critical of the value Gene has offered with these principles.

    • Andrew

      Hi DM, you are certainly a better politician than I am. Though I don’t “want” to be critical, I think the discussion could be most useful if we are presented a view that while acknowledges the ‘potential’ of the ‘Three Ways’, it focuses a bit more on what it is we are trying to ‘transform’, why it is seen as ‘broken’, and what are perhaps the ‘real’ cultural and financial reasons for it being ‘broken’ and in need of ‘transformation’ in the first place.

      DevOps facilitates the process by building (via code) the conduits between the developer and customers through ‘continual integration’; connectivity between ‘monitoring triggers and events’ and the organization’s ‘ticketing’ systems; and advanced configuration management and/or complete ‘containerization’ of deployable environments, among other job descriptors. Operations just needs get on board and/or get out of the way.

      I know a lot about DevOps because that is what I and my company do for many large and small organizations. And, we are paid quite well for our efforts. And, I don’t want to bite the hand that feeds me. But, a lot of what I do is so successful and/or so ‘necessary’ partially due to failed politics and/or the organization’s lacking or non-existent procedures: not necessarily due to the ‘strength’ of the ‘Three Ways’ or any other philosophical sound bites.

      To the extent that “DevOps” can be viewed as ‘getting Operations to be more like Development’, then we understand why the nature of the ‘Service Desk’ has become replaced by on an automated and/or ‘Self-Service’ model. Developers rely on user/customer feedback and restful ‘PUT’ of crash and stack trace output into a ‘continual improvement system’, whereby defects can be detected, presumably corrected, and released back to the customer in record time.

      If only through sheer numbers, and given their equal weight in the discussion, start-ups and smaller companies with fewer products, few, if any ‘mission critical customer services’, fewer processes and employees, who adopt the ‘Agile’ method, can point to quick development and few ‘catastrophic’ failures as proof their ‘Three Ways’ end-to-end, continual feedback, risk taking and reward yielding culture is an unwavering success.

      Larger organizations with significant multiples in the number of products and employees, and a few absolutely ‘mission critical’ services only need to ‘onboard’ this leaner DevOps mentality and their IT will be sufficiently ‘transformed’.

      In the ‘First Way’, I am a strong believer in the IT professional’s need to understand their ‘system’, or ecosystem, from end-to-end. However, as the number of ecosystems grow along with complexity of each ecosystem, the advantage and/or sustainability of any single person or group ‘understanding’ each and every ecosystem at a sufficiently ‘proficient’ level becomes daunting, at best. Thus, larger organizations begin to pare down the expectations as to ‘what to know’, and the ability therefore to determine ‘defects’ or ‘degradation’ in others’ ecosystems becomes haphazard, at best.

      In the ‘Second Way’, a fully staffed and functioning Service Desk can act as a clearinghouse of feedback from systems, developers, testers, operators and end-users. However, the difficulty, both politically and economically, of implementing such a Service Desk continues to be a major impediment to providing timely and accurate feedback for larger and smaller organizations.

      Mostly from the cultural and political spheres: boutique developers and their product managers “can’t be bothered” with complaints and system faults when it is not directly related to their product’s offerings. And, getting ‘proof’ of that relationship is not always: it requires said developers and managers to agree to investigate in an honest and open manner and to admit fault when it is found.

      Thus, the Service Desk’s deciphering and escalating critical issues are often left to those without the proper skill set, knowledge of the many systems, or political authority to execute on their mission, making their ‘lack of effectiveness’ a self-fulfilling prophecy. Again, this lends unfair credence to the notion that the automated and ‘self-service’ model is the only reliable and cost effective feedback model.

      In the ‘Third Way’, I won’t even discuss here whether an effective ‘change management’ system could not only mitigate risk, but could eliminate the need for ‘risk taking’ entirely. I will touch on whether your electronic check deposit should be considered with the same ‘weight’ of service responsibility as, say, your ‘Facebook like’ of your Mexican dinner, or say, a successful Uber reservation for a car service.

      Obviously, being able to discover and overcome the ‘unknown’ in a timely manner requires taking on some risks. But, the risk that some ‘Facebook likes’ are missed is entirely different than the risks that some ‘bank deposits’ aren’t made.

      Not dwelling on exactly what we mean by risk, I’d also argue that, again, quite frankly, only the ‘Developer’ can gain by risk taking. And, this is a cultural phenomenon found in both ‘non-Transformed’ and ‘fully Transformed’ IT shops. More accurately, the risk/reward system is only geared towards rewarding the Developer. The only ‘risk’ Operations can take that doesn’t backfire is to say ‘No’, so she might live to see another day.

      When Developers take risks that fail, it merely appears that something that was previously broken is not fixed; or a new ‘feature’ is not yet available. The Developer will get another crack, or two, at it, or as many as needed until the ‘thingy’ works. Yet, Operations has to deal with the added inconvenience and risks associated with maintaining the faulty product. When Developers finally ‘get it right’, they get all the accolades. All Operations gets is a little less headache and less risk in maintaining the product and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations..

      When Operations takes risks that fail, customers lose confidence in the product, the organization loses business, and people lose their jobs. When Operations takes risks that are successful, the accolades flow to the Developer for ‘coming up with the solution’, and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations.

      Thus, the ‘real’ reason why DevOps is a success, at least partially, is that we are now seeing ‘developers’ in the Operations role. And, now engineering and product managers have a certain vested interest in their ‘Developer Operator’ in a manner they never had before in their ‘System Administrator Operator’. I’d say the only and persistent gap continues to be in rewarding (compensation of) the risk takers: DevOps engineers have fared only marginally better than their Administrator Operator predecessors.

      To conclude, I am all for ‘Agile’ and the ‘Three Ways’. They are exciting and promising ways of approaching IT and Service delivery. However, the day Operators and Service Desk staff are compensated like Developers; and the day the Service Desk is staffed with actual Developers and top-notch ‘DevOps’ engineers: that is the the day the ‘Three Ways’ can be anything other than a quaint notion befitting a startup with few products and no mission critical services.

Sign up for the IT Revolution Newsletter and get our whitepaper,
The Top 11 Things You Need To Know About DevOps.

We won’t share, sell or spam you. –Gene Kim