A Personal Reinterpretation Of The Three Ways

A note from Gene Kim:

Last year, I got an email from Tim Hunter that blew me away.  Among many brilliant things, he sent me a rewrite of The Three Ways, which are intended to be a set of principles from which you can derive all the observed DevOps patterns from.

When I read what he wrote, my jaw dropped.  It was like having a poet rewrite your dissertation thesis.  I’ve incorporated his language of Flow, Feedback and Continual Experimentation and Learning into all my talks now, and it will be incorporated into the next printing of “The Phoenix Project.”

I hope you all enjoy this as much as I did!

Hi! My name is Tim Hunter, known as @Thorrsson on Twitter (and just about everywhere else) and I would like you to join me on a bit of a journey. This journey brings us to my current state in IT and reveals bits about how I came to this reinterpretation of  the Three Ways that Gene Kim and the team on the Phoenix Project defined.

I have worked in various facets of IT since the 90’s, starting off in a dialup ISP, building and selling systems, working with the sysadmins on our internal servers and helping people who had no idea why they wanted to be on the internet get onto the internet. This was my first taste of the Flow of Value. The faster we got our custom built $3000 PCs out the door, the more money we made, etc.

Next I spent a good deal of time working on, and then helping run a call center for a large company. Our ‘value’ was realized at the end of a call when the systems were working again, our Work In Progress was the call itself, we had 3 to 20 minute cycles on average. At one point, the company signed an agreement to support a Food Chain’s Servers and Point of Sale terminals. Apparently, we decided that our SLA was 7 minutes for every call (decided by some sales person).

This should have been OK, as some calls were resolved quickly, others not so much, but we could probably average it out to 7 minutes. Unfortunately the customer had just deployed an extremely buggy back end system ‘upgrade’ that coincided with our takeover of support. Our cycles were averaging 40 to 50 minutes! We had a backlog at almost all times, and there was no way we could average that down to 7 minutes (we eventually got to know the bugs, and could resolve them much faster, but I don’t think we brought our average below 15 minutes).

As a company, we paid penalties like mad for this, and eventually the customer took support back internally. This particular position made me realize that even if your pipelines are sized appropriately, Flow is affected by many different factors.

Now I will jump to the last couple of positions in my career (ya.. there was a bunch between this call center and now, but not as relevant as these next two)

I took a job as a network admin for an automation company — we built robots… (ya I know.. cool isn’t it?) Our robots were self-contained assembly lines most of the time. They had their own networking gear, they had raw material inputs and something cool came out the other end. The really neat part was watching the Toolers and the Developers working together to get the Value Flow just right inside of each of these machines. They would build the machine to spec, write the PLC code to make it do what it needed to, and then they spent a long time tuning the systems to address bottlenecks and constraints, to make sure that the Inputs didn’t back up at any time, and that we had the right input and output rates. Flow and Feedback were king, and no one silo or team was elevated over the other, everyone worked together to get it right.

Next I joined a pure angular development company, a 7 year old start up working in Knowledge Graphs (Interest graphs… 7 years ago!) We had a Disruptive product that already had some time to mature, until the market realized that personalization and interest graphs were going to be our Next Web Technology. Then we needed to increase our Flow rates. It was with this company that I started trying to break down the silos and move past the Way of Flow. When I started Development was Development and Ops were the guys who said ‘No’ and worried about stability. QA was the gateway to OPs and we spent hours troubleshooting deployments. With a lot of hard work from all of our teams we brought deployments down from most of a day to under 30 minutes. (There were some edge cases that were faster or slower… but on average…). We recognized our constraints and elevated them to the best of our abilities, near the end of my time with this company, we had OPs attending the Devs Standups, we (Ops) knew about infrastructure affecting changes before they were checked in, and had an environment ready to support those changes. Our Developers and Business folks were all on the appropriate mailing lists to know about outages when they happened. We were continuously deploying 2 separate environments (Not prod though) and iterating much faster on our experiments.

When the Phoenix Project was about to release, I received the preview of the book from IT Revolution Press, and found myself thinking (on numerous occasions) “I have been Bill in this situation before” or “I remember <Name of Co-Worker> going through something just like this”. A couple of days before the release(if memory serves), Gene gave a Webinar for BMC (and a free Kindle version of the book!), and I came to the realization that the takeaways from this book mirrored things that I had learned along the way, and that I needed to know everything I could about the Three Ways. I stalked Gene for a bit, finding all of the articles I could that referenced the Three Ways. I found presentations he gave at various conferences and articles written for IBM, I even found the skeleton pages for the detailed posts on the Second and Third Ways that had not been written yet :D. Every single one of these rang true, so I reached out to Gene, and introduced myself. Giving a similar, if not more detailed, account of what I have been through in my career and asking if he had anything else on the Three Ways that I could read. After a couple of emails back and forth and I was consuming even more about the Three Ways.

It was at that point that I decided that I needed to socialize this with the company I was working for, and started writing a summary of the Three Ways in my own words, with my own past influencing the descriptions. I iterated on this document for a month or so, coming back to it and tweaking it, and then shared it with Gene, asking if he could sanity check it in case I was off base anywhere. Now, Gene had just released a bestseller, and was running around promoting it, so it took a while before he got to it, though when he did, I found some amazing tweets about the doc (I will just post the public one here):

During this time, I decided to take a new job, and move my family 3800 miles from home, but I still found a bit of time to spend chatting with Gene about this document. We hacked at it a little, and Gene asked if I would like to post it here in a guest blog post, so here we are.

If you made it this far, awesome.. the best part is coming up now. Here is the (barely) modified Three Ways that I sent to Gene in February and we cleaned up a bit in May.

The Three Ways: A Reinterpretation

The Three Ways describe values that define process and procedures as well as interactions. From learning The Three Ways we can adjust our interactions and our philosophies to help the entire organization win.

The First Way: Flow

The First Way is the way of Flow.

It places Dev as the business representative and Ops as the customer representative, with the value flowing in one direction (from the business to the customer). When we can think as a system we can focus clearly on the business value that flows between our Business, Dev, Ops and the end users. We can see each piece as it fits into the whole, and can identify its constraints. We can also properly define our work and when we can see and think in terms of the Flow of our system, we see the following benefits:

• increased value flow due to the visibility into what it takes to produce our end product

• our downstream step always gets what they need, how they need it, when they need it

• faster time to market

• we bring Ops in earlier in the development process, letting them plan appropriately for the changes that Dev will be making (because we know that all changes can affect how our product is delivered) which leads to less unplanned work or rushed changes

• because work is visible, Ops can see the work coming and better prepare

• We can identify and address constraints or bottleneck points in our system

• We don’t let a defect get passed discovery, we ‘stop the line’ and do not push that problem forward.

The Second Way: Feedback

The Second Way is the way of Feedback.

It adds a backward facing channel of communications between OPs and Dev. It enforces the idea that to better the product, we always need to communicate. Dev continually improves as an organization when it better sees the outcomes of it’s work. This can be small (inviting the other Tribes to our stand ups) or it can be larger (Including Dev in the on-call rotation, tools development, architecture planning and/or incident management process) But to truly increase our Flow and improve the business value being delivered to the customer our Tribes need to know ‘what happens’, ‘when it happens’. When we increase our Feedback and create a stable Feedback loop we see the following benefits:

• Tribal knowledge grows, and we foster a community of sharing

• With sharing comes trust and with trust comes greater levels of collaboration. This collaboration will lead to more stability and better Flow

• We better understand all of our customers (Ops as a customer, Dev as a Business, but especially our end users, to whom we deliver value.)

• We fix our defects faster, and are more aware of what is needed to make sure that type of problem doesn’t happen again

• We adapt our processes as we learn more about the inner workings or our other Tribes

• We increase our delivery speeds and decrease unplanned work

The Third Way: Continual Experimentation And Learning

The Third Way is the way of Continual Experimentation and Learning.

When we have achieved the first Two Ways we can feel comfortable knowing that we can push the boundaries. We can experiment, and fail fast, or achieve greatness. We have a constant feedback loop for each small experiment that allows us to validate our theories quickly.

When we increase our Experimentation and Learning we see the following:

• we fail often and sometimes intentionally to learn how to respond properly and where our limits are

• we inject faults into the production system and early as possible in the delivery pipeline

• we practice for outages and find innovative ways to deal with them

• we push ourselves into the unknown more frequently and become comfortable in the uncomfortable

• we innovate and iterate in a ‘controlled’ manner, knowing when should keep pushing and when we should stop

• our code commits are more reliable, and production ready

• we test our business hypotheses (at the beginning of the product pipeline), and measure the business results

• we constantly put pressure into the system, striving to decrease cycle times and improve flow

- About The Authors

Tim Hunter