The Three Ways: The Principles Underpinning DevOps

August 22, 2012 by Gene Kim 36 Comments

In this blog post, I talk about the “Three Ways,” which are the principles that all of the DevOps patterns can be derived from, which we’re using in both the “DevOps Handbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.” We assert that the Three Ways describe the values and philosophies that frame the processes, procedures, practices of DevOps, as well as the prescriptive steps.

It has been especially fun working on this with fellow co-author Mike Orzen, the most recent winner of the cherished Shingo Prize for his contribution to operational excellence for his book “Lean IT.”

The Three Ways are as follows. In a future blog post, I’ll describe the DevOps patterns that can be derived from each.

The First Way emphasizes the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, system administrator).

The focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.

The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system (as per Deming).

The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.

The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.

The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.

We need both of these equally. Experimentation and taking risks are what ensures that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.

The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.

Pingback: Elements Of The First Way: And The DevOps Implications… | IT Revolution()
DM Colburn

I applaud Gene Kim for his contribution to the DevOps transformation effort. These 3 principles offer a valuable focus for evolving the culture change requirements that are the core challenge for embedding a DevOps value system. I do have a suggestion for discussion that may serve to enhance the value of these 3 principles.

The focus on “Systems Thinking” is a concern, if we all agree that the biggest challenge to DevOps is changing current culture, values and rewards thinking. Wouldn’t the focus on developing and managing “systems” support an internal IT-centric perspective? Wouldn’t a focus on Dev and Ops co-designing “Services” better support a business-focused perspective on the value of IT? I submit that integrating the core values of DevOps into the IT Service Lifecycle would most certainly be better served by a ‘service’ focus for the 3 principles.

For internal IT departments to remain relevant there must be a change in how the business values IT. IT must evolve from a ‘cost center’ or ‘spend-only’ department, to a ‘service provider’ or ‘contributing’ department. To be successful IT must be capable of demonstrating value in the same way, and at the same valuation points business departments use to demonstrate their value (i.e. optimizing business process.)

Business department leaders are inherently familiar with and experienced in quantifying (valuing) ‘services’. ‘Systems’ on the other hand, are more
commonly recognized as the purview of IT or manufacturing-related departments. Manufacturing-related departments however, have evolved a more mature capability to quantify their value in support of core business process. IT departments are struggling to evolve this same capability.

DevOps is a critical paradigm shift required for IT to evolve the capability to value its services in a way business department leaders understand. Building serviceability, availability, reliability and service level quantifiability into the business process automation services IT designs, transitions, operates and continually improves, requires integrating development and operations as a single discipline functioning in all phases of the Service Lifecycle.

For this reason I wonder if it would be more ‘IT Transformation’ supportive to include the concept of DevOps co-developing services as a core principal, avoiding the possibility the ‘systems’ perspective in the First Way above, would be seen as perpetuating the internal IT focus for DevOps, as opposed to a business-focused ‘service’ perspective.

My intent is to facilitate discussions on this point, to enhance, not be critical of the value Gene has offered with these principles.
- Andrew
  
  Hi DM, you are certainly a better politician than I am. Though I don’t “want” to be critical, I think the discussion could be most useful if we are presented a view that while acknowledges the ‘potential’ of the ‘Three Ways’, it focuses a bit more on what it is we are trying to ‘transform’, why it is seen as ‘broken’, and what are perhaps the ‘real’ cultural and financial reasons for it being ‘broken’ and in need of ‘transformation’ in the first place.
  
  DevOps facilitates the process by building (via code) the conduits between the developer and customers through ‘continual integration’; connectivity between ‘monitoring triggers and events’ and the organization’s ‘ticketing’ systems; and advanced configuration management and/or complete ‘containerization’ of deployable environments, among other job descriptors. Operations just needs get on board and/or get out of the way.
  
  I know a lot about DevOps because that is what I and my company do for many large and small organizations. And, we are paid quite well for our efforts. And, I don’t want to bite the hand that feeds me. But, a lot of what I do is so successful and/or so ‘necessary’ partially due to failed politics and/or the organization’s lacking or non-existent procedures: not necessarily due to the ‘strength’ of the ‘Three Ways’ or any other philosophical sound bites.
  
  To the extent that “DevOps” can be viewed as ‘getting Operations to be more like Development’, then we understand why the nature of the ‘Service Desk’ has become replaced by on an automated and/or ‘Self-Service’ model. Developers rely on user/customer feedback and restful ‘PUT’ of crash and stack trace output into a ‘continual improvement system’, whereby defects can be detected, presumably corrected, and released back to the customer in record time.
  
  If only through sheer numbers, and given their equal weight in the discussion, start-ups and smaller companies with fewer products, few, if any ‘mission critical customer services’, fewer processes and employees, who adopt the ‘Agile’ method, can point to quick development and few ‘catastrophic’ failures as proof their ‘Three Ways’ end-to-end, continual feedback, risk taking and reward yielding culture is an unwavering success.
  
  Larger organizations with significant multiples in the number of products and employees, and a few absolutely ‘mission critical’ services only need to ‘onboard’ this leaner DevOps mentality and their IT will be sufficiently ‘transformed’.
  
  In the ‘First Way’, I am a strong believer in the IT professional’s need to understand their ‘system’, or ecosystem, from end-to-end. However, as the number of ecosystems grow along with complexity of each ecosystem, the advantage and/or sustainability of any single person or group ‘understanding’ each and every ecosystem at a sufficiently ‘proficient’ level becomes daunting, at best. Thus, larger organizations begin to pare down the expectations as to ‘what to know’, and the ability therefore to determine ‘defects’ or ‘degradation’ in others’ ecosystems becomes haphazard, at best.
  
  In the ‘Second Way’, a fully staffed and functioning Service Desk can act as a clearinghouse of feedback from systems, developers, testers, operators and end-users. However, the difficulty, both politically and economically, of implementing such a Service Desk continues to be a major impediment to providing timely and accurate feedback for larger and smaller organizations.
  
  Mostly from the cultural and political spheres: boutique developers and their product managers “can’t be bothered” with complaints and system faults when it is not directly related to their product’s offerings. And, getting ‘proof’ of that relationship is not always: it requires said developers and managers to agree to investigate in an honest and open manner and to admit fault when it is found.
  
  Thus, the Service Desk’s deciphering and escalating critical issues are often left to those without the proper skill set, knowledge of the many systems, or political authority to execute on their mission, making their ‘lack of effectiveness’ a self-fulfilling prophecy. Again, this lends unfair credence to the notion that the automated and ‘self-service’ model is the only reliable and cost effective feedback model.
  
  In the ‘Third Way’, I won’t even discuss here whether an effective ‘change management’ system could not only mitigate risk, but could eliminate the need for ‘risk taking’ entirely. I will touch on whether your electronic check deposit should be considered with the same ‘weight’ of service responsibility as, say, your ‘Facebook like’ of your Mexican dinner, or say, a successful Uber reservation for a car service.
  
  Obviously, being able to discover and overcome the ‘unknown’ in a timely manner requires taking on some risks. But, the risk that some ‘Facebook likes’ are missed is entirely different than the risks that some ‘bank deposits’ aren’t made.
  
  Not dwelling on exactly what we mean by risk, I’d also argue that, again, quite frankly, only the ‘Developer’ can gain by risk taking. And, this is a cultural phenomenon found in both ‘non-Transformed’ and ‘fully Transformed’ IT shops. More accurately, the risk/reward system is only geared towards rewarding the Developer. The only ‘risk’ Operations can take that doesn’t backfire is to say ‘No’, so she might live to see another day.
  
  When Developers take risks that fail, it merely appears that something that was previously broken is not fixed; or a new ‘feature’ is not yet available. The Developer will get another crack, or two, at it, or as many as needed until the ‘thingy’ works. Yet, Operations has to deal with the added inconvenience and risks associated with maintaining the faulty product. When Developers finally ‘get it right’, they get all the accolades. All Operations gets is a little less headache and less risk in maintaining the product and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations..
  
  When Operations takes risks that fail, customers lose confidence in the product, the organization loses business, and people lose their jobs. When Operations takes risks that are successful, the accolades flow to the Developer for ‘coming up with the solution’, and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations.
  
  Thus, the ‘real’ reason why DevOps is a success, at least partially, is that we are now seeing ‘developers’ in the Operations role. And, now engineering and product managers have a certain vested interest in their ‘Developer Operator’ in a manner they never had before in their ‘System Administrator Operator’. I’d say the only and persistent gap continues to be in rewarding (compensation of) the risk takers: DevOps engineers have fared only marginally better than their Administrator Operator predecessors.
  
  To conclude, I am all for ‘Agile’ and the ‘Three Ways’. They are exciting and promising ways of approaching IT and Service delivery. However, the day Operators and Service Desk staff are compensated like Developers; and the day the Service Desk is staffed with actual Developers and top-notch ‘DevOps’ engineers: that is the the day the ‘Three Ways’ can be anything other than a quaint notion befitting a startup with few products and no mission critical services.
- Erick Hagstrom
  
  You seem to conflate “Systems Thinking” with the “systems” that IT builds, maintains and operates. In fact, “systems thinking” is a discipline completely distinct from IT, and the systems about which we think are frequently human in nature. Wikipedia can give you a good start in understanding the meaning of “Systems Thinking”, though the Waters Foundation does it more succinctly: http://watersfoundation.org/systems-thinking/what/.
Toby Weiss

I noticed “tools” were not one of the ways. Comments anyone? I’ll look for other articles on the tools versus process/culture topic as well.
- Steve Speicher
  
  Tools only help (or hurt) with the 3 principles. If you consider for example the 3rd principle, many customers would look for tools/products/solutions to help with this experimentation and repetitiveness. The simple way I would think of this is by leveraging a simple and fast way to onboard developers, such a Platform-as-a-Service (PaaS) like OpenShift. In addition, it allows for native continuous integration and deployment (repetitive). These can be enhanced by a number of 3rd part systems such as Jenkins, Shippable, ContinuousCI, etc
yunitaasfihani

finally crystal x di jakarta and obat gatal gatal di selangkangan has references about DevOps
- imno haree
  
  These references are only for the advanced. They are highly cryptic and you must provide your credit card details for the secrets of DevOps and other secrets of the universe to exposed to you in all their glory. Please first read the Phoenix Project for amateur standard DevOps before you can decode the black art contained within these links.
pramitasadewi

jual crystal x di depok feel grateful to find an article about this DevOps
- Jason McSpacin
  
  ironically we are not grateful for jual crystal x di depok at all. In fact i’d liken it to SpamOps, which is a very well practiced craft for polluting the internet and trying to sell us crap that would never really be delivered anyway.
dianfebri

This article has given a little more knowledge of the jual crystal x di depok surrounding The Principles underpinning DevOps
- Jock Spicer
  
  i found the link above only gave me the first way and therefore is not true devops.
Evgeny Zislis

I am now reading the first chapter of “The DevOps Toolkit”. And this term that Gene really loves “The Three Ways” appears all over. First, it is not the same as described here – since if I understand correctly System Thinking was changed with Flow.

Here and there in the text appears “First Way”, “Second Way”, “Third Way”, bla bla … I just think it is a very very poor choice to refer to named principles like this. I never remember what does first/second/third mean and it is very distracting. In most of the other books that I actually learned a lot from, there are distinct names for principles – and these names are used and re-used without giving them numbers or letters or whatever, unless their initials fit well into an acronym.

Would you agree that Gene needs to follow each Way Number with its name, or am I just being unreasonable?
- Erick Hagstrom
  
  I agree that “First Way”, etc. is not a very helpful naming convention. It’s like naming your variables A1, A2, etc. (Excuse me a moment while I shudder at the memory of having to produce working code in BASIC.)
  As for the First Way, both the formulations you mention are equivalent. Systems Thinking is all about optimizing the system, streamlining it, making it more reliable, etc. Improving it. What happens when you do those things to a system that manages the “Flow” of work from business needs to services that deliver business value? You end up optimizing the flow, streamlining it, making it more reliable, etc. Improving it. Systems Thinking is how you approach the system, better Flow is the result of improving it.
- davidkallen
  
  I agree. And in fact, I have already begun to translate the “three ways” in my head into the following 3 words he uses for them:Flow, Feedback, Experimentation. But I may modify that based on the excellent comment by @erick_hagstrom:disqus . Maybe I’ll say System Thinking, Feedback, Experimentation. Anything is better than learning a new trinity. That feels like joining a cult. And I don’t like cults. I don’t want to get my teams to drink the coolaid. I want them to think and learn. But I do like the authors and their ideas. So I’m not throwing the baby out with the bathwater. I’m just going to downplay the “Three Ways” thing. Thanks for highlighting this. It was bugging me too, but I had not examined it until you replied.
joeconnors

Thanks for posting and explaining about DevOps in three ways i.e, System thinking, Amplify feedback loops, Culture of continual experimentation and learning.

DevOps is a development methodology with a set of practices aimed at reducing the time between Development and Operations. Now a days DevOps is best future career for students.
donramm

“The Phoenix Project” seems to imply that “security” is nothing but an a roadblock. Might this book have been written a few years ago before the world realized that security has been an afterthought…if any thought was put into it at all?
- Hawk
  
  My impression of the book was that security put efforts on regulatory compliance issues that did not need solving on it side. DevOPS on the other hand encourages that security is everyone’s concern. My impression.
- Uziel
  
  The Phoenix Project framed security not as a set of regulations that needed to follow, but rather a sort of value that needed to be delivered. Security testing and compliance was not a list of boxes to check off, but rather followed the same ideas and processes practiced throughout the book. Those boxes still need to be checked off from a legal standpoint, of course. But from an industry(ial) standpoint, security needs to be treated and tested just like normal code, and is as much a piece of value as the UI is.
martin james

Thanks for your detailed explanation. By going through your post, I have attained a clear knowledge on the end to end aspects of DevOps. Also, looking forward to see more of such informative postings.
For more details on DevOps Training In Hyderabad please visit
http://www.kellytechno.com/Hyderabad/Course/devops-training
- sunjeeva
  
  I did the course being for devops in hyderbad. how do i install the devops, they didnt give me an install executable. please do the needful.
AWS training in hyderabad

Thanks for sharing “The Three Ways: The Principles Underpinning DevOps” very informative post with a good pictorial information.
Peter Row

The biggest challenge no one has mentioned, at least as far as I read in the comments is your customers. It’s all very well being able to iterate quickly, but in my experience customers never want to put that much effort in. They are all about doing the job they are paid to do that just happens to use the software your wrote and provided. We release on a steady scheduled 8 week cycle a new test version and allow 3 weeks for customers to test and feedback, followed by a new build for penetration testing and then a week after an RTM build for production. I can tell you that we are extremely lucky to get more than 1 customer who will actively feedback during the allotted 3 week period despite the dates for all versions within the year being known upfront. Of course when it goes RTM all customers want to install it on their test system first before going live with it.
- Brian Wawok
  
  This is the wonder of SaaS. You just upgrade every day. The user had no choice 🙂 8 week cycles for desktop software is just too slow.
  - Fredrik
    
    …or you upgrade your desktop software automatically, like Google Chrome or Spotify.
    - Peter Row
      
      That’s not the way businesses work. Consumer software like chrome sure, but where you have to upgrade a database before pushing the software and the customer will not accept that in production before they have tried it on their test system it’s a complete other ball gave. I’d love to say rough just have the latest and we’ll release a new version to fix any issues is simply not something that business will accept.
      - Ted Husted
        
        A lot of businesses today who buy into a SAAS model do accept it.
        
        Take Salesforce.com for example. Major upgrades are pushed three times a year, period. Patch upgrades are deploy at least once a week. As far as I know, there is no way to opt out.
        
        Migrating existing products to a SAAS model when there is already low-trust could certainly be a problem, but a great many businesses will and do accept the model, when presented properly.
        
        As the original poster said, customers never want to put that much effort in — but trust is essential.
      - Peter Row
        
        I guess these people have never worked in local government before then.
      - Raju
        
        I agree to Peter. Other than core IT businesses like salesforce, chrome etc, there is lot of resistance from customers for every week or every month upgrade. Idealistically, process works but practically challenges in government are numerous
      - Aaron Surty
        
        I completely agree. I wish there was a handbook for ISVs writing enterprise software for on premise use.
  - Peter Row
    
    It is web software not desktop. For desktop software it would be mega fast. You are thinking of things like a browser, but business customers and business software there is no way they will go that fast.
    - jer
      
      As a SaaS provider telcos also have issues accepting direct to live software changes. not least as no SaaS testing is done in the same e2e environment and where the client is the ultimate responsible of the service level to the end customer. What ends up happening is continuous testing that has significant extra costs.
- Brian Turnwald
  
  I’ve found the best way to get feedback to to proactively ask for it in person or on a call. When I email or put something “out there” and ask them to test, it never gets done. Customers love to provide feedback, but don’t make them work for it. Make it easy!
Vaes Theo

Thank you for sharing this model. I will adapt it for our social innovation project to learn resolving generational poverty in a durable manner. Armentekort.be
Gary Williams

This article annoys me. Why? Because it’s 100% right and yet devops today is associated with automation, the cloud and “devops engineers”. Not one of them understands the three ways.
santhosh kumar

We are grepthor software solutions in hyderabad. We have been dealing many IT projects along with digital marketing projects. We give best service to the clients.

The Three Ways: The Principles Underpinning DevOps

Featured Book

Events