In this blog post, I talk about the “Three Ways,” which are the principles that all of the DevOps patterns can be derived from, which we’re using in both the “DevOps Handbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.” We assert that the Three Ways describe the values and philosophies that frame the processes, procedures, practices of DevOps, as well as the prescriptive steps.
It has been especially fun working on this with fellow co-author Mike Orzen, the most recent winner of the cherished Shingo Prize for his contribution to operational excellence for his book “Lean IT.”
The Three Ways are as follows. In a future blog post, I’ll describe the DevOps patterns that can be derived from each.
The First Way emphasizes the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, system administrator).
The focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.
The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system (as per Deming).
The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.
The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.
We need both of these equally. Experimentation and taking risks are what ensures that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.
The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.
I applaud Gene Kim for his contribution to the DevOps transformation effort. These 3 principles offer a valuable focus for evolving the culture change requirements that are the core challenge for embedding a DevOps value system. I do have a suggestion for discussion that may serve to enhance the value of these 3 principles.
The focus on “Systems Thinking” is a concern, if we all agree that the biggest challenge to DevOps is changing current culture, values and rewards thinking. Wouldn’t the focus on developing and managing “systems” support an internal IT-centric perspective? Wouldn’t a focus on Dev and Ops co-designing “Services” better support a business-focused perspective on the value of IT? I submit that integrating the core values of DevOps into the IT Service Lifecycle would most certainly be better served by a ‘service’ focus for the 3 principles.
For internal IT departments to remain relevant there must be a change in how the business values IT. IT must evolve from a ‘cost center’ or ‘spend-only’ department, to a ‘service provider’ or ‘contributing’ department. To be successful IT must be capable of demonstrating value in the same way, and at the same valuation points business departments use to demonstrate their value (i.e. optimizing business process.)
Business department leaders are inherently familiar with and experienced in quantifying (valuing) ‘services’. ‘Systems’ on the other hand, are more
commonly recognized as the purview of IT or manufacturing-related departments. Manufacturing-related departments however, have evolved a more mature capability to quantify their value in support of core business process. IT departments are struggling to evolve this same capability.
DevOps is a critical paradigm shift required for IT to evolve the capability to value its services in a way business department leaders understand. Building serviceability, availability, reliability and service level quantifiability into the business process automation services IT designs, transitions, operates and continually improves, requires integrating development and operations as a single discipline functioning in all phases of the Service Lifecycle.
For this reason I wonder if it would be more ‘IT Transformation’ supportive to include the concept of DevOps co-developing services as a core principal, avoiding the possibility the ‘systems’ perspective in the First Way above, would be seen as perpetuating the internal IT focus for DevOps, as opposed to a business-focused ‘service’ perspective.
My intent is to facilitate discussions on this point, to enhance, not be critical of the value Gene has offered with these principles.
Hi DM, you are certainly a better politician than I am. Though I don’t “want” to be critical, I think the discussion could be most useful if we are presented a view that while acknowledges the ‘potential’ of the ‘Three Ways’, it focuses a bit more on what it is we are trying to ‘transform’, why it is seen as ‘broken’, and what are perhaps the ‘real’ cultural and financial reasons for it being ‘broken’ and in need of ‘transformation’ in the first place.
DevOps facilitates the process by building (via code) the conduits between the developer and customers through ‘continual integration’; connectivity between ‘monitoring triggers and events’ and the organization’s ‘ticketing’ systems; and advanced configuration management and/or complete ‘containerization’ of deployable environments, among other job descriptors. Operations just needs get on board and/or get out of the way.
I know a lot about DevOps because that is what I and my company do for many large and small organizations. And, we are paid quite well for our efforts. And, I don’t want to bite the hand that feeds me. But, a lot of what I do is so successful and/or so ‘necessary’ partially due to failed politics and/or the organization’s lacking or non-existent procedures: not necessarily due to the ‘strength’ of the ‘Three Ways’ or any other philosophical sound bites.
To the extent that “DevOps” can be viewed as ‘getting Operations to be more like Development’, then we understand why the nature of the ‘Service Desk’ has become replaced by on an automated and/or ‘Self-Service’ model. Developers rely on user/customer feedback and restful ‘PUT’ of crash and stack trace output into a ‘continual improvement system’, whereby defects can be detected, presumably corrected, and released back to the customer in record time.
If only through sheer numbers, and given their equal weight in the discussion, start-ups and smaller companies with fewer products, few, if any ‘mission critical customer services’, fewer processes and employees, who adopt the ‘Agile’ method, can point to quick development and few ‘catastrophic’ failures as proof their ‘Three Ways’ end-to-end, continual feedback, risk taking and reward yielding culture is an unwavering success.
Larger organizations with significant multiples in the number of products and employees, and a few absolutely ‘mission critical’ services only need to ‘onboard’ this leaner DevOps mentality and their IT will be sufficiently ‘transformed’.
In the ‘First Way’, I am a strong believer in the IT professional’s need to understand their ‘system’, or ecosystem, from end-to-end. However, as the number of ecosystems grow along with complexity of each ecosystem, the advantage and/or sustainability of any single person or group ‘understanding’ each and every ecosystem at a sufficiently ‘proficient’ level becomes daunting, at best. Thus, larger organizations begin to pare down the expectations as to ‘what to know’, and the ability therefore to determine ‘defects’ or ‘degradation’ in others’ ecosystems becomes haphazard, at best.
In the ‘Second Way’, a fully staffed and functioning Service Desk can act as a clearinghouse of feedback from systems, developers, testers, operators and end-users. However, the difficulty, both politically and economically, of implementing such a Service Desk continues to be a major impediment to providing timely and accurate feedback for larger and smaller organizations.
Mostly from the cultural and political spheres: boutique developers and their product managers “can’t be bothered” with complaints and system faults when it is not directly related to their product’s offerings. And, getting ‘proof’ of that relationship is not always: it requires said developers and managers to agree to investigate in an honest and open manner and to admit fault when it is found.
Thus, the Service Desk’s deciphering and escalating critical issues are often left to those without the proper skill set, knowledge of the many systems, or political authority to execute on their mission, making their ‘lack of effectiveness’ a self-fulfilling prophecy. Again, this lends unfair credence to the notion that the automated and ‘self-service’ model is the only reliable and cost effective feedback model.
In the ‘Third Way’, I won’t even discuss here whether an effective ‘change management’ system could not only mitigate risk, but could eliminate the need for ‘risk taking’ entirely. I will touch on whether your electronic check deposit should be considered with the same ‘weight’ of service responsibility as, say, your ‘Facebook like’ of your Mexican dinner, or say, a successful Uber reservation for a car service.
Obviously, being able to discover and overcome the ‘unknown’ in a timely manner requires taking on some risks. But, the risk that some ‘Facebook likes’ are missed is entirely different than the risks that some ‘bank deposits’ aren’t made.
Not dwelling on exactly what we mean by risk, I’d also argue that, again, quite frankly, only the ‘Developer’ can gain by risk taking. And, this is a cultural phenomenon found in both ‘non-Transformed’ and ‘fully Transformed’ IT shops. More accurately, the risk/reward system is only geared towards rewarding the Developer. The only ‘risk’ Operations can take that doesn’t backfire is to say ‘No’, so she might live to see another day.
When Developers take risks that fail, it merely appears that something that was previously broken is not fixed; or a new ‘feature’ is not yet available. The Developer will get another crack, or two, at it, or as many as needed until the ‘thingy’ works. Yet, Operations has to deal with the added inconvenience and risks associated with maintaining the faulty product. When Developers finally ‘get it right’, they get all the accolades. All Operations gets is a little less headache and less risk in maintaining the product and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations..
When Operations takes risks that fail, customers lose confidence in the product, the organization loses business, and people lose their jobs. When Operations takes risks that are successful, the accolades flow to the Developer for ‘coming up with the solution’, and the financial and hierarchical benefits likewise, flow to the Development groups and rarely to Operations.
Thus, the ‘real’ reason why DevOps is a success, at least partially, is that we are now seeing ‘developers’ in the Operations role. And, now engineering and product managers have a certain vested interest in their ‘Developer Operator’ in a manner they never had before in their ‘System Administrator Operator’. I’d say the only and persistent gap continues to be in rewarding (compensation of) the risk takers: DevOps engineers have fared only marginally better than their Administrator Operator predecessors.
To conclude, I am all for ‘Agile’ and the ‘Three Ways’. They are exciting and promising ways of approaching IT and Service delivery. However, the day Operators and Service Desk staff are compensated like Developers; and the day the Service Desk is staffed with actual Developers and top-notch ‘DevOps’ engineers: that is the the day the ‘Three Ways’ can be anything other than a quaint notion befitting a startup with few products and no mission critical services.
You seem to conflate “Systems Thinking” with the “systems” that IT builds, maintains and operates. In fact, “systems thinking” is a discipline completely distinct from IT, and the systems about which we think are frequently human in nature. Wikipedia can give you a good start in understanding the meaning of “Systems Thinking”, though the Waters Foundation does it more succinctly: http://watersfoundation.org/systems-thinking/what/.
I noticed “tools” were not one of the ways. Comments anyone? I’ll look for other articles on the tools versus process/culture topic as well.
Tools only help (or hurt) with the 3 principles. If you consider for example the 3rd principle, many customers would look for tools/products/solutions to help with this experimentation and repetitiveness. The simple way I would think of this is by leveraging a simple and fast way to onboard developers, such a Platform-as-a-Service (PaaS) like OpenShift. In addition, it allows for native continuous integration and deployment (repetitive). These can be enhanced by a number of 3rd part systems such as Jenkins, Shippable, ContinuousCI, etc
finally crystal x di jakarta and obat gatal gatal di selangkangan has references about DevOps
These references are only for the advanced. They are highly cryptic and you must provide your credit card details for the secrets of DevOps and other secrets of the universe to exposed to you in all their glory. Please first read the Phoenix Project for amateur standard DevOps before you can decode the black art contained within these links.
jual crystal x di depok feel grateful to find an article about this DevOps
ironically we are not grateful for jual crystal x di depok at all. In fact i’d liken it to SpamOps, which is a very well practiced craft for polluting the internet and trying to sell us crap that would never really be delivered anyway.
This article has given a little more knowledge of the jual crystal x di depok surrounding The Principles underpinning DevOps
i found the link above only gave me the first way and therefore is not true devops.
I am now reading the first chapter of “The DevOps Toolkit”. And this term that Gene really loves “The Three Ways” appears all over. First, it is not the same as described here – since if I understand correctly System Thinking was changed with Flow.
Here and there in the text appears “First Way”, “Second Way”, “Third Way”, bla bla … I just think it is a very very poor choice to refer to named principles like this. I never remember what does first/second/third mean and it is very distracting. In most of the other books that I actually learned a lot from, there are distinct names for principles – and these names are used and re-used without giving them numbers or letters or whatever, unless their initials fit well into an acronym.
Would you agree that Gene needs to follow each Way Number with its name, or am I just being unreasonable?
I agree that “First Way”, etc. is not a very helpful naming convention. It’s like naming your variables A1, A2, etc. (Excuse me a moment while I shudder at the memory of having to produce working code in BASIC.)
As for the First Way, both the formulations you mention are equivalent. Systems Thinking is all about optimizing the system, streamlining it, making it more reliable, etc. Improving it. What happens when you do those things to a system that manages the “Flow” of work from business needs to services that deliver business value? You end up optimizing the flow, streamlining it, making it more reliable, etc. Improving it. Systems Thinking is how you approach the system, better Flow is the result of improving it.
I agree. And in fact, I have already begun to translate the “three ways” in my head into the following 3 words he uses for them:Flow, Feedback, Experimentation. But I may modify that based on the excellent comment by @erick_hagstrom:disqus . Maybe I’ll say System Thinking, Feedback, Experimentation. Anything is better than learning a new trinity. That feels like joining a cult. And I don’t like cults. I don’t want to get my teams to drink the coolaid. I want them to think and learn. But I do like the authors and their ideas. So I’m not throwing the baby out with the bathwater. I’m just going to downplay the “Three Ways” thing. Thanks for highlighting this. It was bugging me too, but I had not examined it until you replied.
Scientists have no trouble referring to Newton’s 3 laws of motion directly by number. Surely us business and tech people that have port numbers and command line syntax and acronyms memorized can remember the three ways to accomplish the Goal.
It took me a bit to learn them. The book that The Phoenix Project is based on, The Goal by Eliyahu M. Goldratt, helped a great deal. It is a bit corny at times… but it’s a great way to solidify the concepts of the three ways.
“The Phoenix Project” seems to imply that “security” is nothing but an a roadblock. Might this book have been written a few years ago before the world realized that security has been an afterthought…if any thought was put into it at all?
My impression of the book was that security put efforts on regulatory compliance issues that did not need solving on it side. DevOPS on the other hand encourages that security is everyone’s concern. My impression.
The Phoenix Project framed security not as a set of regulations that needed to follow, but rather a sort of value that needed to be delivered. Security testing and compliance was not a list of boxes to check off, but rather followed the same ideas and processes practiced throughout the book. Those boxes still need to be checked off from a legal standpoint, of course. But from an industry(ial) standpoint, security needs to be treated and tested just like normal code, and is as much a piece of value as the UI is.
The biggest challenge no one has mentioned, at least as far as I read in the comments is your customers. It’s all very well being able to iterate quickly, but in my experience customers never want to put that much effort in. They are all about doing the job they are paid to do that just happens to use the software your wrote and provided. We release on a steady scheduled 8 week cycle a new test version and allow 3 weeks for customers to test and feedback, followed by a new build for penetration testing and then a week after an RTM build for production. I can tell you that we are extremely lucky to get more than 1 customer who will actively feedback during the allotted 3 week period despite the dates for all versions within the year being known upfront. Of course when it goes RTM all customers want to install it on their test system first before going live with it.
This is the wonder of SaaS. You just upgrade every day. The user had no choice 🙂 8 week cycles for desktop software is just too slow.
…or you upgrade your desktop software automatically, like Google Chrome or Spotify.
That’s not the way businesses work. Consumer software like chrome sure, but where you have to upgrade a database before pushing the software and the customer will not accept that in production before they have tried it on their test system it’s a complete other ball gave. I’d love to say rough just have the latest and we’ll release a new version to fix any issues is simply not something that business will accept.
A lot of businesses today who buy into a SAAS model do accept it.
Take Salesforce.com for example. Major upgrades are pushed three times a year, period. Patch upgrades are deploy at least once a week. As far as I know, there is no way to opt out.
Migrating existing products to a SAAS model when there is already low-trust could certainly be a problem, but a great many businesses will and do accept the model, when presented properly.
As the original poster said, customers never want to put that much effort in — but trust is essential.
I guess these people have never worked in local government before then.
I agree to Peter. Other than core IT businesses like salesforce, chrome etc, there is lot of resistance from customers for every week or every month upgrade. Idealistically, process works but practically challenges in government are numerous
I completely agree. I wish there was a handbook for ISVs writing enterprise software for on premise use.
It is web software not desktop. For desktop software it would be mega fast. You are thinking of things like a browser, but business customers and business software there is no way they will go that fast.
As a SaaS provider telcos also have issues accepting direct to live software changes. not least as no SaaS testing is done in the same e2e environment and where the client is the ultimate responsible of the service level to the end customer. What ends up happening is continuous testing that has significant extra costs.
I’ve found the best way to get feedback to to proactively ask for it in person or on a call. When I email or put something “out there” and ask them to test, it never gets done. Customers love to provide feedback, but don’t make them work for it. Make it easy!
Thank you for sharing this model. I will adapt it for our social innovation project to learn resolving generational poverty in a durable manner. Armentekort.be
This article annoys me. Why? Because it’s 100% right and yet devops today is associated with automation, the cloud and “devops engineers”. Not one of them understands the three ways.
I just transitioned away from a “DevOps Engineer” role because not only was it not these three ways, it wasn’t even your “automation, the cloud…”, it was just Operations++
“operations++” – I like that phrase. I hope you don’t mind if I steal it for use in meetings?
Operations++? So like a Site Reliability Engineer just getting started, before they start statistically analyzing log data? 🙂
That reeducation is part of the journey. I’m constantly having to show that it is a cultural and technical mindset, not a specific role or tool.
That’s mainly because people associate the buzzword with concepts they’ve been fed by marketers, not by practitioners in the field. Uncertain yet if MS renaming their collaboration and work delivery tool as Azure DevOps is going to hurt or to help matters. So far it’s at least seemed like a boon to my quest to spread these practices and concepts where I work.
Btw, this page has been one of my startup tabs for longer than I can remember. It regrounds me in the concepts I need to use in the coming day. Probably the closest I’ve gotten to meditation anytime recently.
I don’t see why DevOps would exclude proactive automation and simplification. Peter Merel calls these The Zeroth Way Of DevOps.
Six years late to the party, but… are these really three “ways” or three layers/steps? Three ways imply three *different* paths to getting to the same thing.
I’m also conflicting a bit with that issue. English is not my native language, perhaps it makes sense for a English native?
They are three steps. Without thinking about the 1st way, the 2nd way can lead to local optimizations that don’t have an important impact on the whole system. The continual experimentation and learning of the 3rd way requires the shorter feedback loops of the 2nd way. It’s something that’s meant to be thought of in steps for sure.
I’m reading the Phoenix Project now and find this fascinating. I’ve had execs asking me about implementing “DevOps tools and processes” and I keep telling them that DevOps is more about adopting and committing to a culture and set of values than just tools and processes to add to your disfunctional IT area. The “3 ways” are presented in such a lofty and philosophical way, like guidance from a Kung Fu master, which makes it difficult to sell to corporate execs who want to check boxes and measure the results of precise action plans rather than change their thinking.
I really like the Capabilities to Drive Improvement section in Accelerate (https://itrevolution.com/book/accelerate/). It outlines exactly which practices folllow the 3 ways and are shown to improve IT performance.
Thanks, for the article Gene. IMHO there is still huge human factor impact here you cannot well automate..
Natalie Sanelyte | Digital Reviewer | easybuilder.pro
Thanks for the information regarding Devops
Amazing write up, never read such an informative blog and enjoyed it. Thankyou. Keep up the good work. Looking forward to read more.
it is written very nicely and is optimized