DevOps Blog

Get the RSS Feed

The Three Ways: The Principles Underpinning DevOps

In this blog post, I talk about the “Three Ways,” which are the principles that all of the DevOps patterns can be derived from, which we’re using in both the  “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.” We assert that the Three Ways describe the values and philosophies that frame the processes, procedures, practices of DevOps, as well as the prescriptive steps.

It has been especially fun working on this with fellow co-author Mike Orzen, the most recent winner of the cherished Shingo Prize for his contribution to operational excellence for his book “Lean IT.”

The Three Ways are as follows.  In a future blog post, I’ll describe the DevOps patterns that can be derived from each.

The First Way emphasizes the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, system administrator).

The focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.

The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system (as per Deming).

The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.

The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.

The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.

We need both of these equally. Experimentation and taking risks are what ensures that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.

The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.

  • Pingback: Elements Of The First Way: And The DevOps Implications… | IT Revolution()

  • DM Colburn

    I applaud Gene Kim for his contribution to the DevOps transformation effort. These 3 principles offer a valuable focus for evolving the culture change requirements that are the core challenge for embedding a DevOps value system. I do have a suggestion for discussion that may serve to enhance the value of these 3 principles.

    The focus on “Systems Thinking” is a concern, if we all agree that the biggest challenge to DevOps is changing current culture, values and rewards thinking. Wouldn’t the focus on developing and managing “systems” support an internal IT-centric perspective? Wouldn’t a focus on Dev and Ops co-designing “Services” better support a business-focused perspective on the value of IT? I submit that integrating the core values of DevOps into the IT Service Lifecycle would most certainly be better served by a ‘service’ focus for the 3 principles.

    For internal IT departments to remain relevant there must be a change in how the business values IT. IT must evolve from a ‘cost center’ or ‘spend-only’ department, to a ‘service provider’ or ‘contributing’ department. To be successful IT must be capable of demonstrating value in the same way, and at the same valuation points business departments use to demonstrate their value (i.e. optimizing business process.)

    Business department leaders are inherently familiar with and experienced in quantifying (valuing) ‘services’. ‘Systems’ on the other hand, are more
    commonly recognized as the purview of IT or manufacturing-related departments. Manufacturing-related departments however, have evolved a more mature capability to quantify their value in support of core business process. IT departments are struggling to evolve this same capability.

    DevOps is a critical paradigm shift required for IT to evolve the capability to value its services in a way business department leaders understand. Building serviceability, availability, reliability and service level quantifiability into the business process automation services IT designs, transitions, operates and continually improves, requires integrating development and operations as a single discipline functioning in all phases of the Service Lifecycle.

    For this reason I wonder if it would be more ‘IT Transformation’ supportive to include the concept of DevOps co-developing services as a core principal, avoiding the possibility the ‘systems’ perspective in the First Way above, would be seen as perpetuating the internal IT focus for DevOps, as opposed to a business-focused ‘service’ perspective.

    My intent is to facilitate discussions on this point, to enhance, not be critical of the value Gene has offered with these principles.

    • Andrew

      Hi DM, you are certainly a better politician than I am. Though I don’t “want” to be critical, I think the discussion could be most useful if we are presented a view that while acknowledges the ‘potential’ of the ‘Three Ways’, it focuses a bit more on what it is we are trying to ‘transform’, why it is seen as ‘broken’, and what are perhaps the ‘real’ cultural and financial reasons for it being ‘broken’ and in need of ‘transformation’ in the first place.

      DevOps facilitates the process by building (via code) the conduits between the developer and customers through ‘continual integration’; connectivity between ‘monitoring triggers and events’ and the organization’s ‘ticketing’ systems; and advanced configuration management and/or complete ‘containerization’ of deployable environments, among other job descriptors. Operations just needs get on board and/or get out of the way.

      I know a lot about DevOps because that is what I and my company do for many large and small organizations. And, we are paid quite well for our efforts. And, I don’t want to bite the hand that feeds me. But, a lot of what I do is so successful and/or so ‘necessary’ partially due to failed politics and/or the organization’s lacking or non-existent procedures: not necessarily due to the ‘strength’ of the ‘Three Ways’ or any other philosophical sound bites.

      To the extent that “DevOps” can be viewed as ‘getting Operations to be more like Development’, then we understand why the nature of the ‘Service Desk’ has become replaced by on an automated and/or ‘Self-Service’ model. Developers rely on user/customer feedback and restful ‘PUT’ of crash and stack trace output into a ‘continual improvement system’, whereby defects can be detected, presumably corrected, and released back to the customer in record time.

      If only through sheer numbers, and given their equal weight in the discussion, start-ups and smaller companies with fewer products, few, if any ‘mission critical customer services’, fewer processes and employees, who adopt the ‘Agile’ method, can point to quick development and few ‘catastrophic’ failures as proof their ‘Three Ways’ end-to-end, continual feedback, risk taking and reward yielding culture is an unwavering success.

      Larger organizations with significant multiples in the number of products and employees, and a few absolutely ‘mission critical’ services only need to ‘onboard’ this leaner DevOps mentality and their IT will be sufficiently ‘transformed’.

      In the ‘First Way’, I am a strong believer in the IT professional’s need to understand their ‘system’, or ecosystem, from end-to-end. However, as the number of ecosystems grow along with complexity of each ecosystem, the advantage and/or sustainability of any single person or group ‘understanding’ each and every ecosystem at a sufficiently ‘proficient’ level becomes daunting, at best. Thus, larger organizations begin to pare down the expectations as to ‘what to know’, and the ability therefore to determine ‘defects’ or ‘degradation’ in others’ ecosystems becomes haphazard, at best.

      In the ‘Second Way’, a fully staffed and functioning Service Desk can act as a clearinghouse of feedback from systems, developers, testers, operators and end-users. However, the difficulty, both politically and economically, of implementing such a Service Desk continues to be a major impediment to providing timely and accurate feedback for larger and smaller organizations.

      Mostly from the cultural and political spheres: boutique developers and their product managers “can’t be bothered” with complaints and system faults when it is not directly related to their product’s offerings. And, getting ‘proof’ of that relationship is not always: it requires said developers and managers to agree to investigate in an honest and open manner and to admit fault when it is found.

      Thus, the Service Desk’s deciphering and escalating critical issues are often left to those without the proper skill set, knowledge of the many systems, or political authority to execute on their mission, making their ‘lack of effectiveness’ a self-fulfilling prophecy. Again, this lends unfair credence to the notion that the automated and ‘self-service’ model is the only reliable and cost effective feedback model.

      In the ‘Third Way’, I won’t even discuss here whether an effective ‘change management’ system could not only mitigate risk, but could eliminate the need for ‘risk taking’ entirely. I will touch on whether your electronic check deposit should be considered with the same ‘weight’ of service responsibility as, say, your ‘Facebook like’ of your Mexican dinner, or say, a successful Uber reservation for a car service.

      Obviously, being able to discover and overcome the ‘unknown’ in a timely manner requires taking on some risks. But, the risk that some ‘Facebook likes’ are missed is entirely different than the risks that some ‘bank deposits’ aren’t made.

      Not dwelling on exactly what we mean by risk, I’d also argue that, again, quite frankly, only the ‘Developer’ can gain by risk taking. And, this is a cultural phenomenon found in both ‘non-Transformed’ and ‘fully Transformed’ IT shops. More accurately, the risk/reward system is only geared towards rewarding the Developer. The only ‘risk’ Operations can take that doesn’t backfire is to say ‘No’, so she might live to see another day.

      When Developers take risks that fail, it merely appears that something that was previously broken is not fixed; or a new ‘feature’ is not yet available. The Developer will get another crack, or two, at it, or as many as needed until the ‘thingy’ works. Yet, Operations has to deal with the added inconvenience and risks associated with maintaining the faulty product. When Developers finally ‘get it right’, they get all the accolades. All Operations gets is a little less headache and less risk in maintaining the product and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations..

      When Operations takes risks that fail, customers lose confidence in the product, the organization loses business, and people lose their jobs. When Operations takes risks that are successful, the accolades flow to the Developer for ‘coming up with the solution’, and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations.

      Thus, the ‘real’ reason why DevOps is a success, at least partially, is that we are now seeing ‘developers’ in the Operations role. And, now engineering and product managers have a certain vested interest in their ‘Developer Operator’ in a manner they never had before in their ‘System Administrator Operator’. I’d say the only and persistent gap continues to be in rewarding (compensation of) the risk takers: DevOps engineers have fared only marginally better than their Administrator Operator predecessors.

      To conclude, I am all for ‘Agile’ and the ‘Three Ways’. They are exciting and promising ways of approaching IT and Service delivery. However, the day Operators and Service Desk staff are compensated like Developers; and the day the Service Desk is staffed with actual Developers and top-notch ‘DevOps’ engineers: that is the the day the ‘Three Ways’ can be anything other than a quaint notion befitting a startup with few products and no mission critical services.

    • Erick Hagstrom

      You seem to conflate “Systems Thinking” with the “systems” that IT builds, maintains and operates. In fact, “systems thinking” is a discipline completely distinct from IT, and the systems about which we think are frequently human in nature. Wikipedia can give you a good start in understanding the meaning of “Systems Thinking”, though the Waters Foundation does it more succinctly:

  • Toby Weiss

    I noticed “tools” were not one of the ways. Comments anyone? I’ll look for other articles on the tools versus process/culture topic as well.

    • Steve Speicher

      Tools only help (or hurt) with the 3 principles. If you consider for example the 3rd principle, many customers would look for tools/products/solutions to help with this experimentation and repetitiveness. The simple way I would think of this is by leveraging a simple and fast way to onboard developers, such a Platform-as-a-Service (PaaS) like OpenShift. In addition, it allows for native continuous integration and deployment (repetitive). These can be enhanced by a number of 3rd part systems such as Jenkins, Shippable, ContinuousCI, etc

  • yunitaasfihani

    finally crystal x di jakarta and obat gatal gatal di selangkangan has references about DevOps

  • pramitasadewi

    jual crystal x di depok feel grateful to find an article about this DevOps

  • dianfebri

    This article has given a little more knowledge of the jual crystal x di depok surrounding The Principles underpinning DevOps

  • Evgeny Zislis

    I am now reading the first chapter of “The DevOps Toolkit”. And this term that Gene really loves “The Three Ways” appears all over. First, it is not the same as described here – since if I understand correctly System Thinking was changed with Flow.

    Here and there in the text appears “First Way”, “Second Way”, “Third Way”, bla bla … I just think it is a very very poor choice to refer to named principles like this. I never remember what does first/second/third mean and it is very distracting. In most of the other books that I actually learned a lot from, there are distinct names for principles – and these names are used and re-used without giving them numbers or letters or whatever, unless their initials fit well into an acronym.

    Would you agree that Gene needs to follow each Way Number with its name, or am I just being unreasonable?

    • Erick Hagstrom

      I agree that “First Way”, etc. is not a very helpful naming convention. It’s like naming your variables A1, A2, etc. (Excuse me a moment while I shudder at the memory of having to produce working code in BASIC.)
      As for the First Way, both the formulations you mention are equivalent. Systems Thinking is all about optimizing the system, streamlining it, making it more reliable, etc. Improving it. What happens when you do those things to a system that manages the “Flow” of work from business needs to services that deliver business value? You end up optimizing the flow, streamlining it, making it more reliable, etc. Improving it. Systems Thinking is how you approach the system, better Flow is the result of improving it.

    • davidkallen

      I agree. And in fact, I have already begun to translate the “three ways” in my head into the following 3 words he uses for them:Flow, Feedback, Experimentation. But I may modify that based on the excellent comment by @erick_hagstrom:disqus . Maybe I’ll say System Thinking, Feedback, Experimentation. Anything is better than learning a new trinity. That feels like joining a cult. And I don’t like cults. I don’t want to get my teams to drink the coolaid. I want them to think and learn. But I do like the authors and their ideas. So I’m not throwing the baby out with the bathwater. I’m just going to downplay the “Three Ways” thing. Thanks for highlighting this. It was bugging me too, but I had not examined it until you replied.

  • joeconnors

    Thanks for posting and explaining about DevOps in three ways i.e, System thinking, Amplify feedback loops, Culture of continual experimentation and learning.

    DevOps is a development methodology with a set of practices aimed at reducing the time between Development and Operations. Now a days DevOps is best future career for students.

  • donramm

    “The Phoenix Project” seems to imply that “security” is nothing but an a roadblock. Might this book have been written a few years ago before the world realized that security has been an afterthought…if any thought was put into it at all?

Sign up for the IT Revolution Newsletter and get our whitepaper,
The Top 11 Things You Need To Know About DevOps.

We won’t share, sell or spam you. –Gene Kim