One of the best validations about the way we characterized the downward IT spiral in the book is the number of people leaving Amazon comments similar to this: 

• “I find myself relating to the characters in The Phoenix Project and as others have commented – I’ve probably met most of them over the course of my career.”
• “If you have ever worked in and aspect of IT, DevOps, or Infosec you will definitely be able to relate to situations in this book.”
• “There’s not a character in The Phoenix Project that I don’t identify with myself or someone I know in real life… not to mention the problems faced and overcome by those characters.”

(I’m also delighted that, at the time of this writing, Amazon shows 102 5-star reviews!  Thank you, all!)

I’ll show one PowerPoint slide that describes the downward spiral, and how to use it to help sell the value of the DevOps to others.  (Of course, this technique is equally applicable for all the other concepts in the book, including top-down view of risk and infosec, kanbans, elevation of preventive work, two week improvement cycles, etc.)

In a future post, I will discuss the steps in the Logical Thinking Processes, how we created the characters which model many of the IT archetypes and roles, and the various software tools I used during the writing process.

(The downward IT spiral is also the basis of the talk I put together for DevOpsDays London called “How To Better Sell DevOps (slideshare link)” which I did at the urging of Jonathan Thrope (@jonathan_thorpe’s blog) and John Clapham (@JohnC_Bristol) over breakfast.)

Why A Novel? Storytelling Is The Most Effective Means Of Creating A Shared Understanding

For many challenges in life, whether it’s to get a project approved, persuade them to your point of view, plan a family vacation, earn someone’s trust, or ask someone out on a date, our goal is to persuade and to get someone else’s mirror neurons to fire.

Studies have repeatedly shown that the most effective mechanism for persuasion is storytelling. I remember reading somewhere this is because the human brain is structured to hear stories. Joseph Campbell’s concept of “The Hero’s Journey” shows how epic tales and legends all share similar patterns, and Kurt Vonnegut has his unforgettable “Simple Shapes Of Stories” concept (my favorite being this 5 minute video where he shows the graph of famous stories).

You find the Hero’s Journey pattern everywhere: it’s the core of the solution-selling pattern, good copywriting, and even the best TED talks: they describe the problem, show its significance, describe how the problem is solved and the value it will creates.

Storytelling becomes very important when the problem we’re trying to point out is not fully recognized, or when the solution being proposed flies in the face of common wisdom. It’s been my thesis that the problems that motivate DevOps is not only the most important problem facing IT, but is also the most important business problem that must be solved. However, it’s a business problem that is not widely recognized or even believed, and even the believers (myself included) often have difficulty localizing or precisely verbalizing it.

When I first read “The Goal” in 1998, it completely blew my mind. Even though I’ve never worked in a manufacturing plant, let alone managed one, it was amazing to me to see the world through Alex Rogo’s eyes as he had to fix his cost and quality issues in 90 days, otherwise his world, and I knew that the lessons being taught in the book were important and relevant to my professional career.

It is widely cited “The Goal” influenced an entire generation of professional plant managers around the world. This was documented in an appendix in the third revised edition of “The Goal.” However, I believe the best description of this phenomena is described on Disc 6 of Dr. Goldratt’s audio set “Beyond The Goal”, where he described the letters he received. The letters would almost always say something like, “You have obviously been hiding in our factory, because you’ve described my life [as a plant manager] exactly…”

I mentioned in a previous blog post how we had been preparing to write “The Phoenix Project” for nearly ten years. We wanted to write “The Goal” for the modern IT context, because we kept seeing the same problems over and over, and we believed that everyone needed to see that when IT fails, the business fails. And that if all the various IT stakeholders (e.g., Dev, IT Ops, Infosec), as well as “The Business” and even Audit, could work together, the business could win and win big.

The Structure Of “The Goal”

It’s difficult to overstate how much we consciously mirrored the structure of “The Goal.” Having studied that book over a decade, here’s how I deconstruct the book:

• Part 1: vivid description of the problem (170 pages)
  • Verbalization of predominant beliefs (e.g., cost accounting, needs for “efficiency”, large batch sizes, etc.)
  • Depiction of how these beliefs and resulting practices pre-ordain failure (e.g., orders never on time, customers never receive what they need, plant is unprofitable, layoffs, plant is in danger of being shutdown, Alex on verge of being fired, etc.)
• Part 2: iterative solving of problems in 90 days (the rest of the book)
  • The search for the constraint (e.g., heat treat ovens and NCX-10)
  • Breakthrough 1: Creating red/green tags for parts heading towards the constraints (subordinate)
  • Breakthrough 2: Ensuring continuous three shifts of operations of the constraint
  • Breakthrough (exploit)
  • Breakthrough 3: Find outsourcers/vendors to expand constraint capacity (elevate)
  • Breakthrough 4: Reduce batch sizes by 50% to alleviate WIP at non-constraints (implementing drum-buffer-rope pattern)
  • Breakthrough 5: Finding additional sales demand when plant capacity exceeds market demand (constraint moves outside of plant)

(Thanks to Dr. James Holt of Washington State University for his two graduate courses on Constraints Management and all of his “kitchen table coaching,” as well as to Dr. Scott R. Schultz at Mercer University for his excellent synopsis of the book.)

The IT Downward Spiral As The First 170 Pages Of “The Phoenix Project”

Dr. Goldratt stated in “Beyond The Goal” that “not even one description or hint of what the solution to Alex’s problems were even mentioned until page 170.” We took that concept very literally, and baked that structure into “The Phoenix Project.”

The first 170 pages of “The Phoenix Project” is the narrative form of over fourteen years of research into the downward spiral that IT and the business undergo when Development, IT Operations and Infosec don’t work work together to achieve the global goals of the organization, and when Infosec is not properly subordinated into the flow of work.

For years, the selling tool for DevOps was the one PowerPoint slide shown below, which describes the downward IT spiral that occurs when Dev and IT Operations don’t work together (it is slide 11 in this presentation).

Why was this slide so effective? Because you can walk someone through the descriptions of the Ops issues, and then the Dev issues, and then ask, “does this problem resonate with you at all?”

And the response is almost always, “Holy crap. That’s us.” Which inevitably leads to the question, “How can you help us escape this downward spiral?”  (Someone once said to me, ”Before I can trust someone, I first need to know they care [about what I care about.]“)

Why? Because this phenomena occurs in every IT organization. It is an inevitable consequence of the core, chronic conflict in IT: in order to help the business win, every IT organization must respond to urgent business needs (i.e., make changes more quickly), while providing reliable, stable and secure IT service (i.e., make changes more slowly, if ever).

(I include this table in text form at the end of the blog post.)

Describing The Downward Spirals As A Mechanism To Better Sell DevOps

Downward spirals are bad because they are positively reinforcing feedback loops. In other words, left unchecked, the problems get worse. There are a several downward spirals in the slide:

• fragile artifacts become more fragile
• technical debt grows
• date-driven application projects focus only on features, sacrificing non-functional requirements, which results in more fragile artifacts in production
• application deploy take longer, become more turbulent, and continually gets worse
• IT Ops is mired in firefighting, and therefore cannot do preventive work or new projects
• long feature delivery cycle times result in more political decision making, meaning more focus on features (vs. non-functional requirements)

But the most important point is the business implications: when Dev and IT Operations miss their commitments, the company misses the commitment it made to the outside world. This affects not only the CIO, but also their boss, and their boss’s boss.

And that’s why this one slide has been so effective: it describes the local Dev and IT Operations problems, but also shows how it almost preordains failure of the global goals, which the CEO and the board need to care about.

(After all, 95% of all capital projects have some IT reliance, and 50% of all capital spending is technology-related. As Chris Little (@BMC_DevOps) said so pithily, “Every business, regardless of what business they think they’re in, is an IT company.”)

The slide doesn’t show the Infosec problems, who get aced out of the game by the Dev / IT Operations tribal warfare. But Infosec contributes to this core, chronic conflict (I’ll explain this concept later in the post), too:

• Infosec feeds endless amounts of production vulnerabilities (sometimes unneeded) to IT Operations
• Patches to fragile production systems cause outages
• IT Operations becomes less likely to deploy patches
• Development uses up all the time in the schedule, leaving no time for security testing
• Infosec project reviews jeopardize due date and cost targets
• Only Infosec work performed is audit and compliance related (lowest nutritional value)
• Infosec allows unnecessary and unneeded audit remediation work to be performed by Development and IT Operations
• Infosec continually becomes less and less integrated into daily work of Development and IT Operations
• Infosec “find to fix” time gets worse and worse
• Infosec increasingly unable to help the organization achieve its goals

When walking people through these points, the goal is to get the other person (regardless of their role) to say, “Holy cow. You understand my problems and these problems are important to me.”

And in the ideal, their next question will be, “How can you help me?”

Resulting Structure For “The Phoenix Project” (Warning: some spoilers enclosed)

As described earlier, we knew we had 170 pages to describe the problem. But, what is the scope of the problem being described?

I’ve found that one of the most useful constructs for describing business goals is the COSO Enterprise Risk Management Cube (or as we practitioners call it, “the COSO Cube”) construct by the COSO Commision. Here’s my favorite, which is version 2. (Yes, even cubes have versions.

The top of the COSO Cube describes the four objectives that every organization has in order to achieve its global goals: strategy, accurate financial reporting, compliance with laws and regulations, and operations.

• Strategy: as embodied by the Phoenix Project, which the entire future of the company depends upon, requires that IT be a core competency.
• Accurate financial reporting: as embodied by the third year repeat audit findings around the IT general controls, but also the loss of accounts payable and inventory management systems which prevents the closing the financial books at the end of quarter. Even the payroll failure resulted in inaccurate payroll numbers, which led to financial reporting errors.
• Compliance with laws and regulations (e.g., SOX-404, PCI DSS): failing the SOX-404 audit results in an adverse footnotes by the external auditor in the SEC 10-K statement, and there are grave implications for failing the PCI DSS assessment.
• Operations (e.g., IT application and project delivery, IT operations, information security): the Project Phoenix was $20 million over-budget and three years late, and when it finally was deployed into production, everyone would have been better off it hadn’t. Furthermore, virtually all of the critical business systems that run daily operations require that IT services be running correctly, which they often weren’t.

(I’m laughing as I write this, because the situation that poor Bill must face in “The Phoenix Project” really is the perfect storm, as it shows all four of the COSO Cube internal control objectives being jeopardized. But, the Amazon reviews show that almost every company is at risk of being like Parts Unlimited. The difference is only in degree.)

So, the structure for Part 1 of “The Phoenix Project” became:

• Part 1: vivid description of the problem (170 pages)
  • Verbalization of observations of current reality (e.g., fragile artifacts, production failures, too many audit findings, technical debt, IT production issues resulting in jeopardizing all the COSO internal control objectives)
  • Depiction of how predominant practices pre-ordain failure (e.g., IT Operations measured mostly on uptime and availability, Development and Product Management measured primarily on features and time to market, which trump IT Operations concerns, Infosec measured on compliance and injecting work into the IT system)

Similarly, we constructed the remaining portions of “The Phoenix Project” to be a series of eight breakthroughs as Bill and team figured out how to identify, exploit, subordinate and elevate the constraint:

• Breakthrough 1: discovering too much WIP in IT Operations and gobsmacking amounts of reliance on Brent for both project and recovery work
• Breakthrough 2: elevating preventive work to prevent unplanned work (especially for Brent) and making work visible
• Breakthrough 3: throttling the flow of work into IT Operations by the project freeze (subordinate)
• Breakthrough 4: identifying and constraining the flows of work to Brent (it was genuinely surprising to find out after the fact at how much this resembles the green/red tag pattern used in “The Goal”)
• Breakthrough 5: documenting and defining standardized work, and managing handoffs to reduce time spent in queue (through use of kanbans), and further reducing reliance on Brent
• Breakthrough 6: correctly identifying reliance on IT systems to hit Dick’s corporate objectives (via Gartner RVM model) and correctly scoping audit and infosec (via GAIT and GAIT-R)
• Breakthrough 7: building DevOps flow of work to better design in non-functional requirements in Dev, reduce batch sizes and enable single-piece flow through Development and IT Operations and better enable operational resilience (replicating the work described by Jez Humble, David Farley, Paul Hammond and John Allspaw)
• Breakthrough 8: when they discover the constraint moves outside the organization, they bring back those resources in-house (replicating one of my favorite Theory of Constraints plays: when there’s idle plant capacity, it’s often cheaper to fabricate parts in-house than to pay a supplier. Lovely.)

In a future blog post, I’ll describe how we used Dr. Goldratt’s Logical Thinking Processes to create the downward IT spiral slide, and how we used the Current Reality Tree to create the book outline.

Please let me know what you think!

The post The Construction Of “The Phoenix Project:” Using The Downward Spiral To Better Sell DevOps appeared first on IT Revolution.

In this post, I will describe the six steps we went through to use Google AdWords to test titles/subtitles for “The Phoenix Project” book, how we tested with specific audiences as defined used the UX-style personas for each of the major characters in the book, how we interpreted the AdWords campaign results, and our conclusions and final decision on the book title.

From start to finish, the testing campaign took three days, cost us about $500, and delivered to us data that gave us confidence to make our final title change to the book.

Why We Urgently Needed Title/Subtitle Testing

For those of you who followed the development of “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win,” you may know that we’ve changed the title several times over the past two years.  Even before Summer 2011, the working titles included:

• “When IT Fails: The Novel”
• “When IT Fails: A Business Novel”
• “When IT Fails: A DevOps Novel”

It’s October 23rd, 2012 and we are having a full-day get together of the publishing team, which consisted of Todd Sattersten, Tim Grahl, Hannah Concannon and me. Among many other urgent things, we still had to finalize the cover design within a month.  And yet, here we were, still arguing about the what the book title should be.

If we didn’t decide on it soon, there was no way we’d have book covers ready, which means that we wouldn’t be able to initiate our first hardcover book print run, which would definitely make it impossible to hit our planned January 15th, 2013 book launch.Several folks whose opinions I trust expressed concerns that the book title with the word “fail” just didn’t seem quite right. Others also said that if we wanted our book to be a shorthand way to refer to a growing revolution in IT, the title needed to be more positive.

Out of growing desperation and a need to make a decision, we finally decided to do what any hypothesis-driven practitioner would do: we decided use Google AdWords to do a title/subtitle testing campaign.

I believe this technique is especially relevant to the DevOps community for the following reasons:

• Some of the tenets of DevOps include using short cycle times to enable iterative development, as well as continuous feedback and testing: techniques like this should be used at the very front-end of the feature funnel, to ensure that there is actual market demand before Development even starts working on it, in conjunction with continuous A/B testing to figure out how to maximize the desired business results.
• We discovered some surprising things about our community of DevOps and IT practitioners: specifically, we found more signs of burnout among the Infosec and ITIL practitioners, revealed by how they voted on the candidate titles.I’ll share what these differences are, as well as my theories on why this is the case… and why Infosec and ITIL practitioners need DevOps the most.

Why Use Google AdWords?

Eric Reis of Lean Startup fame writes: “Our goal [in using Google AdWords] is to find out whether customers are interested in your product by offering to give (or even sell) it to them.”

The underpinning theory is that if no one is willing to click on a Google ad for your new book, software service or offering, they will be incredibly unlikely to click on it on Amazon or buy it when they see if on a bookshelf.  You can read his entire tutorial of how to do a Google AdWords campaign for a new potential offering here.

(Historians will undoubtedly note that the theory and practice of methodical testing before committing capital was described by Tim Ferriss in 2007 in his book “Four Hour Work Week”, using both eBay and Google AdWords.  There are many reasons to follow his works — as my good friend William Hertling once said, “When in doubt, copy Tim Ferriss.”)

So here’s how we set up our experiment.

Step 1: Select Two Titles To Test, Using A Placeholder Subtitle

Our goal in the first step was to find a winning title. For now, we would use a placeholder subtitle, so that we only changed one variable at a time.

One of the titles was the incumbent, “When IT Fails.” For the second title, we chose “The Phoenix Project,” which in the novel, was the project name of the most critical company initiative, but was over two years late and over $20 million over budget.

We chose this because almost every one of the early reviewers would mention the project, alongside all the characters. For this reason, we thought it was a promising and suitable candidate title.

We then went into the Google AdWords campaign management management page, and set up two campaigns. Each campaign has two components: the ad text and ad subtext.

• Title Campaign 1: “When IT Fails: Creating Breakthrough Performance”
• Title Campaign 2: “The Phoenix Project: Creating Breakthrough Performance”

We chose “Creating Breakthrough Performance” as the subtitle, because it offered an attractive (if vague) promise that would hopefully entice people to click on the ad.

The sample Google AdWords ads that would get shown to the right of the Google search results are shown below:

When someone clicked on the ad, they would go to a special landing page we created on the IT Revolution site, thanking them for their feedback.

The next step was to target the Google AdWords campaign (i.e., who would Google put the ads in front of?). For that, we needed to define our audiences.

Step 2: Create Personas From The Novel Characters To Create Our Campaign Search Terms

A common theme in the early feedback we got developing the book was that the characters vividly reminded the readers of people they’ve worked with in their past (or in some cases, people they’re currently working with).

There’s a reason for this. Each of the main characters are a synthesis of the types of people we’ve encountered over the past 10+ years. The characters are like user personas in UX design, personifying the archetypal roles, temperaments and personalities that we’ve all seen in IT organizations.

Listed below are the novel characters, as well as the search terms we thought they’d use in Google:

• Bill (VP of IT Operations): “The Goal”
• Chris (VP of Application Development): “agile,” “scrum,” “Death March,” “The Deadline”
• Patty (Director of IT Service Support): “ITIL,” “ITSM,” “service management”
• John (CISO): “it audit,” “pci compliance,” “cobit,” “infosec,” “information security”
• Erik (the “Jonah”/”Yoda” character): “lean,” “lean IT,” “theory of constraints”

Note that we left out any mention of “Visible Ops” or the author names as search terms, because we only wanted to test the influence of the title and avoid the introduction of any bias in the experiment.

We then configured campaigns for each of the personas with the expected search terms, and tested the two titles against them, shown at the right.

Make sense? To find what the Chrises of the world thought, we configured Google to test our ads to anyone who searched for “agile,” “scrum,” and so forth. To find out what the Johns of the world thought, Google would display our ads for people who search for “PCI compliance,” etc.

Shown at the right are all the campaigns that we created.

Work required to date: four hours; Cost to date: $0.00

Having completed this, we were ready to make the ads live.

Step 3: Analyze And Interpret Results From Title Testing

After running the campaign for one entire day, our title testing campaign served nearly 150K impressions, with 82 click-throughs, costing $170 to execute. The average click-through rate was low (less than 0.2%), but everyone assured us that this expected, given the non-specific nature of the ad.

Here’s what the campaign screen looked like, just for your information — don’t bother trying to read it. We put it down just so you can see what a live campaign looks like. Our analysis follows.

Who Clicked Through By Search Term

The top four search terms resulting in the most clicks came from the following:

• “The Goal”: 31 clicks
• “Agile”: 10 clicks
• “ITIL”: 9 clicks
• “IT Audit”: 7 clicks

Our interpretation of this is that people who were searching for the famous book, “The Goal” by Dr. Eliyahu Goldratt were most likely to click on the ad, because it somehow resonated with them. Given how often the people who gave us feedback made that comparison, that was good news.

The next most frequent search terms were were developers, ITIL practitioners and auditors.

Which Title Won?

Across the entire population, “The Phoenix Project” beat “When IT Fails” by about 2:1. However, it wasn’t uniform. Some personas voted very differently.

Here is the graph of the voting results, broken down by persona:

On the one hand, we can see that people who match the Erik and Chris personas decisively favored “The Phoenix Project.” On the other hand, the Patty persona was split, but the John persona was decisively favoring “When IT Fails.”

When I saw these results, I suddenly was reminded of the study that Josh Corman did on information security burnout. The study noted that signs of burnout included cynicism, fatigue and illusions of self-efficacy. As Josh often quips, “Cynicism isinfosec’s core competency.” (Har har, right?)

I think that cynicism shows in how they voted! Although not nearly as decisively, the ITIL practitioners also voted this way, contrasting with Development.

My belief is that anyone who is forced to live with the downstream consequences (e.g., defects, technical debt) of people made upstream in the value chain (i.e., Development), eventually feel powerless and pre-ordained to failure. This is one of the most damaging things we can do to other people as fellow human beings, and of course causes burnout.

If we had more time, I’m sure that if we could have figured out how to better target IT Operations, they would have voted very similarly to Infosec.

Total elapsed time: 1 day; Total cost to date: $170.

Step 4: Create Campaign To Test Subtitles Alongside Our Winning Title

At this point, we felt confident that “The Phoenix Project” was our winning title. The next step was to come up with potential subtitles. There were about eight subtitles that we created, along the following lines:

• “A Novel About IT, DevOps And Breakthrough Business Performance”
• “A Novel About IT, DevOps And Helping The Business Win”
• “A Novel About IT, DevOps And Creating Breakthrough Performance”

Here’s what they looked like in the Google AdWords campaign screen.

Step 5: Analyze Results From Subtitle Testing

We ran this campaign for about three days, which generated ~218K impressions across 45 keywords. We got 86 clicks on nine of those keywords (“the goal,” “ITIL,” “service management,” “IT audit,” “agile,”) with an average of $2.25 per click.

Some surprises before we got to the analysis phase:

• The high cost per click (CPC) for those keywords: I guess the surprise here was that CPC relative to the cost of the Kindle book ($9.95) indicated that using Google AdWords probably wasn’t going to be cost effective for the book.
• Many of the ads that we configured were being turned off by AdWords due to low quality. This was interesting, as it made sure the ineffective subtitles were being winnowed out of the candidate pool. (We also believe that our overly simple landing page might have triggered this. Todd added more words to the landing page to conviced AdWords that we were legit. Were we to do this again, I’d probably seek out the help of an AdWords pro It might be good to consult with an AdWords expert to make sure we weren’t shooting ourselves in the foot, were we to do this again.)
• Again, “The Goal” and ITIL had some of the highest click-through rates (CTR). The top search terms were: the goal (24), itil (17), service management (13), it audit (12), agile (12).

Here were the results of the voting, broken out by personas:

Note how the Infosec persona was the only population that didn’t like “Helping The Business Win!” I mean, it got less than 5% of the votes, where it was a clear winner for the Erik and ITIL practitioners! More indication of cynacism and burnout?  I don’t know, but it’s fascinating to see how differently they voted.

Step 6: Select And Tune The Winning Subtitle

With these results in hand, Todd Sattersten and I got on Skype and studied the vote tallies, confident that we had narrowed the field down to the best candidates.

After some discussion, we decided to go with the “Helping The Business Win” variant.  We did make a one-word change: we changed the “The Business” to “Your Business”. Why? We wanted to eradicate the horrible habit that we IT practitioners often have, calling the organizations that we serve as “the business,” as if we existed completely outside of it.

(Why does that bother me?  We’re a part of the team, too! We’re not a vendor or a service provider! As Bob Lewis writes in his excellent “Keep The Joint Running” newsletters, acting merely like a utility or a service provider instead of a member of the team helping the organization win is the quickest route to being outsourced,)

I wish I had better documented the other reasons why we chose the “Winning” title, however emotional they were. I think I liked it best, because it best conjured up the vision of everyone in IT being part of a team or a tribe, with a sense of shared values and purpose, working together to help the organization win is really the greatest hope I have for everyone in IT.

In our manifesto, we wrote that we want to “help improve the livelihoods of 1 million IT workers by 2017.” That’s not about creating “breakthrough business performance,” that’s about being part of a tribe that is all helping our businesses and organizations win.

So, there you have it: this is how we did the title/subtitle testing for “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.” Having made that decision, we then gave directions to the book cover artist, which shows all the characters in the book positioned around a bubbling volcano of servers, which represents the Phoenix Project.

I’ll write later on the cover design process, which began before we nailed down the title, and how this title/subtitle testing process influenced the final book cover.

Next up

• Next on the backlog, I’ll be writing about more resources and underpinning theories in “The Phoenix Project:” including GAIT, the Gartner RVM model, kanbans, etc.
• How we designed and constructed the novel, and how the Tinderbox software package saved me when mindmaps, outliners, Scrivener and Evernote couldn’t.

Last note: I see that I started this blog article in December 2012. It’s been sitting on the kanban as “Doing” for 3 months, half written for nearly 60 days! Bad Gene! Too much work in process, WIP is truly the silent killer. I obviously need to adjust down the WIP limit in the my Doing lane in Leankit Kanban. 3 months cycle time for 2500 word article? Unacceptable! Live and learn

The post How We Used Google AdWords For Title/Subtitle Testing For “The Phoenix Project” appeared first on IT Revolution.

One of the most requested items has been a list of recommended reading and further resources to learn more about the philosophies, tools and techniques that were used in the book.

This will be the first in a series of blog posts describing the “body of knowledge” that underpins the novel.  In future blog posts, my goal will be to describe the actual planning and construction of the novel, how I became trained as a Theory of Constraints Jonah, a Kanban practitioner, an acolyte in the Toyota Kata, how Visible Ops is woven into the book, as well as more concrete practice aids to replicate the transformation that Bill pulled off with Erik’s help at Parts Unlimited.

After posting this, I realized that I’ve forgotten several key concepts:

• The GAIT and GAIT-R methodologies (Institute of Internal Auditors) and the Risk-Adjusted Value Management (Gartner): these are what explain John’s seemingly miraculous transformation as a CISO.
• Two books on kanbans, which completely changed the way I visualize and manage work

I’ll write a Part 2 to this post very soon.

Dr. Eliyahu Goldratt’s Theory of Constraints

Dr. Eliyahu Goldratt wrote his seminal book, “The Goal: A Process Of Ongoing Improvement” in 1984.  It’s a Socratic novel about Alex Rogo, a plant manager who must fix his cost and due date issues in ninety days, or his plant will be shut down.  This book has been incorporated into many MBA curriculums and has influenced multiple generations of business leaders, and has sold over 6 million copies to date.

My co-authors and I studied this book for nearly a decade, getting ready to write “The Phoenix Project.”  In many ways, I view our book as an homage to “The Goal.” We attempted to mirror most of the book structure and plot elements, while making it contemporary, relevant, and hopefully more dramatic.  (I’d like to think that “The Phoenix Project” is what Dr. Goldratt would have written if he wrote “The Goal” today, and had Tarantino or Scorsese as a novel coach  :)

In “The Goal,” Dr. Goldratt starts to describes the steps in the Theory of Constraints (TOC) methodology, which I’ll write more about in a future blog post.  Briefly, the five orginial TOC steps are:

• Identify the constraint
• Exploit the constraint
• Subordinate all other activities to the constraint
• Elevate the constraint to new levels
• Find the next constraint

In “The Goal,” the constraints were initially the famous NCX-10 robot, then the heat treat ovens, and then the constraint becomes market demand.  In “The Phoenix Project,” the constraint was initially Brent because he was always having to deal with unplanned work, then the application deployment process, and then the constraint moved outside the organization because the needed MRP application support was outsourced.

In Dr. Goldratt’s following book, “It’s Not Luck,” he describes what he called the Thinking Processes, which is a fantastic (but somewhat inaccessible and often very slow) methodology of identifying core, chronic conflicts, methods of capturing the current reality, describing the desired future reality, and various planning techniques to increase the likelihood of getting there successfully.

By far, the very best overview of of the entire TOC, the Thinking Processes, and Goldratt’s body of knowledge is an audiobook called “Beyond The Goal”.  It includes all of his recorded lectures from 2005, and is a breathtaking tour of Dr. Goldratt’s life journey, describing his contributions, tools, and case studies.

Eight years ago, it was well-known that the Thinking Processes were the tools and techniques he used to construct “The Goal.”  That’s why my co-authors and I tried to learn more about it, but couldn’t find any training or books beyond Dr. Goldratt’s writings that were feasible for us to attend or buy.  Nor were there any non-trivial (or correct) examples that we could find scouring the Internet. Much has changed since then.

For anyone interested in becoming expert in TOC and the Thinking Processes, I wholehearted recommend Dr. James Holt’s EM526 Constraints Management and EM530 Applications Of Constraints Management courses, offered online through Washington State University.  They both prepare students to become trained as “a Jonah,” the Yoda-like character in “The Goal,” who was obviously the embodiment of Dr. Goldratt.  (“Jonah trained” is actually an outdated term.  Official certification programs now exist through TOCICO, the Theory of Constraints International Certification Organization.)

George Spafford and I both took Dr. Holt’s courses and loved them.  You not only gain an understanding of TOC and the Thinking Processes, but it also is a fantastic primer on plant floor planning and operations.  For those of you who are interested in this course, EM526 is currently being taught as a Spring 2013 class.  Maybe it’s not too late for you to register and attend by emailing him — Dr. Holt’s contact information is at the top of the course description page!

(If you can’t take Dr. Holt’s course and really want to learn more about the Thinking Processes, I’d recommend Dr. Dettmer’s textbook, “The Logical Thinking Process”, although it is not light reading.  And many shortcuts have been developed that haven’t been reflected in Dr. Dettmer’s book, so danger lies in this path.)

If you’re interested in Dr. Goldratt’s work, I’d first recommend reading “The Goal,” and then listening to “Beyond The Goal.”  And if you are still interested in learning more, take Dr. Holt’s fantastic courses.  I hope you enjoy them as much as I did.

5 Dysfunctions Of A Team

The technique that Steve, the Parts Unlimited CEO, uses in Chapter 18 in Part 2 is modeled after Patrick Lencioni’s methodology described in “The Five Dysfunctions of a Team: A Leadership Fable.”  He posits that one of the core contributors to a team’s inability to achieve goals is due to lack of trust.  In his model, the five dysfunctions are described as:

• Absence of trust—unwilling to be vulnerable within the group
• Fear of conflict—seeking artificial harmony over constructive passionate debate
• Lack of commitment—feigning buy-in for group decisions creates ambiguity throughout the organization
• Avoidance of accountability—ducking the responsibility to call peers on counterproductive behaviour which sets low standards
• Inattention to results—focusing on personal success, status and ego before team success

When I think about the long, bitter intertribal warfare that has existed between Development and IT Operations, as well as between IT and “the business,” I suspect that we will very much need the lessons of Mr. Lencioni to achieve the DevOps ideal.

Often, the first step in using Mr. Lencioni’s methodology is for leaders to enable themselves to become vulnerable (or at the very least, start by modeling vulnerable behaviors).  In “The Phoenix Project,” Steve has already internalized these practices for decades, and leads what is called a personal history exercise.

I was fortunate enough to have personally observed and benefited from his techniques being used.  When my old boss, Jim B. Johnson, used this technique when he first joined as CEO of Tripwire, Inc., frankly it blew my mind.  He shared his own personal story, which was so personal and touching, it left the rest of us on the executive team emotionally raw, with tears in (almost) everyone’s eyes.

In turn, we all had to share some elements of our own personal story, showing vulnerability to each other, and enabling the next step, which is to stop fearing conflict.  Jim set the tone of the honesty and candor he demanded from everyone, and trust me, it changed how we behaved, executed and we started acting more like a team.

This was probably one of the most important lessons in my life.  It is now my aspiration in every domain of my life to never fear conflict, never be afraid to tell the truth, and never be afraid to say what I really think.  Of course, I’d be delusional to think I can fully achieve this, but I think it’s still a worthy goal.

I’ve been in situations where I’ve observed leadership teams locked in chronic underperformance and strife, because of the utter inability for the team members to trust each other.  And when leaders don’t trust each other, then almost certainly, their respective teams won’t trust each other.

From my professional experience, the cost and true consequence of not being able to have candid discussions about problems that everyone knows about, but is unwilling to confront is incredibly high.  Tackling this problem requires overcoming some of our most ingrained and learned behaviors, but the rewards are worth it.

Toyota Kata

Another one of the books that deeply influenced us is “Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results” by the Shingo Prize winner Mike Rother.  (The Shingo Prize is often called the “Nobel Prize For Manufacturing.  Mr. Rother has been awarded this prize three times.)

I’ve had the pleasure of meeting him, as well as taking his three day course, “Improvement Kata and Coaching Kata,” offered through University of Michigan.  It includes two days of fieldwork in a real manufacturing plant — I’ll be posting my review of this course sometime soon.

If I could take the liberty of describing Mr. Rother’s journey in my own words, it began over twenty years ago as he was visiting Toyota plants with a team of researchers and American car manufacturing executives.  He describes that stage of his journey as capturing and codifying the observed Toyota practices that led to their extraordinary and market-leading performance.

However, when he looks back at that phase of his career,  he’ll characterize it as merely teaching people how to mimic the behaviors observed at the Toyota plants, but missing the most important parts of the Toyota culture and values that framed their management practices.

Listening to him talk about this, it’s as if he was calling a significant portion of the Lean community that he helped train merely a “cargo cult.”  (This term refers to how pre-industrial tribes behaved in New Guinea and Micronesia shortly after World War II. Having observed an influx of material goods brought by American and Japanese soldiers, these tribes were baffled when the war and supply deliveries ended.  In an effort to bring back the deliveries, the tribes replicated the observed behaviors of the U.S. Corp of Engineers, building imitation runways and radio equipment. Despite all of this, for obvious reasons, the planes never returned.)

Mr. Rother’s learnings have been codified in the book, “Toyota Kata,” which frames the thought process and culture that must exist to enable the Lean PDCA cycle (Plan, Do, Check, Act).  I believe that this is one of the most extraordinary contributions to the world of process improvement.

The most obvious manifestation of the Toyota Kata is the “two week improvement cycle,” in which every work center supervisor must improve something (anything!) every two weeks.  To quote Mr. Rother, “The practice of kata is the act of practicing a a pattern so it becomes second nature.  In its day-to-day management, Toyota teaches a way of working — a kata — that has helped make ti so successful over the last six decades.”

The notion of the need for daily repetition in order to create habits, in order to change the outcomes is now well-established in the domains of sports training, learning to play a musical instrument, how Special Forces in the military trains, and now in modern manufacturing.

This forms the basis of Erik’s Third Way that is “about creating a culture that fosters two things:  continual experimentation, which requires taking risks and learning from success and failure; and understanding that repetition and practice is the prerequisite to mastery.”

In my mind, Patty’s ITIL/ITSM crusade is very much like the Lean practitioners that Mr. Rother describes who were never able to replicate the performance of Toyota.  Why?  They’d do a Lean Kaizan event once per year, but  then get marginalized from daily operations the remainder of the year.

For us to get the performance gains promised by ITIL/ITSM, Lean, or whatever, we must create a culture of relentless improvement described by Mr. Rother.

Kata impacts your organization by:

• Providing a systematic, scientific routine that can be applied to any problem or challenge.
• Commonizing how the members of an organization develop solutions.
• Migrating managers toward a role of coach and mentor, by having them practice coaching cycles.
• Framing PDCA in a way that has people taking small steps every day.

These two week improvement cycles put constant pressure into the system, forcing it to improve.  Mr. Rother asserts that if a system is not improving, the result is not a steady state.  Instead, because of entropy, organizational performance declines.

In one of the most startling case studies, Mr. Rother describes observing how a certain work center that was able to decrease the number of workers from six to four.  Over the next six weeks, however, the number of workers gradually grew back to six.  Entropy.

In my mind, patterns like the Netflix culture that has created relentless improvement and innovation, ruthless eradication of variance, and injecting faults into the production environment (embodied in tools such as the famous Chaos Monkey) is the perfect day embodiment of the Improvement Kata that Mr. Rother describes.

Continuous Delivery

Erik’s First Way underscores the importance of “the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, system administrator).”

In the IT value stream, this is all about the left to right flow of work from Development into IT Operations.  Probably the best embodiment of this work is Jez Humble  and David Farley’s seminal book, “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.”

They codify many of the techniques required to replicate the famous 2009 Velocity Conference presentation,“10 Deploys A Day” given by John Allspaw and Paul Hammond, as well as the Agile system administration movement, described further here by John Willis.

Continuous delivery is the extension of continuous integration, which are the Development practices that include continuous builds, continuous testing, daily integration of branches back into trunk, testing in a clone of the production environment, etc.  Continuous delivery techniques extend these processes all the way into the production environment.

(When I read this book, I almost doubled over in pain, realizing how much pain and poor decisions I could have averted if I had read this book four years prior.  I was associated with an unnamed software company where we had a broken build system for over a year.  Without automated builds, you can’t have automated testing, and without automated testing, you’re doomed to painful integration of developer branches, which leads to a downward spiral of less frequent and far more painful integrations, which slows down feature delivery and reduces software quality.)

Imagine my delight when Jez Humble told me that the exercise that the value stream map exercise that Bill and team went through in Chapter 31 of the Development to IT Operations production deployment process was almost precisely what he and Farley did in their book. Awesome.

Continuous delivery is the perfect embodiment of the First, Second and Third Ways, as it emphasizes small batch sizes (e.g., check into trunk daily), stopping the line when problems occur (e.g., no new work allowed when builds, tests or deployments fail; elevating the integrity of the system of work over the work itself), and the need to continually build the validation tests necessary to either prevent failures in production, or at the very least, detect and correct them quickly (e.g., the transition from manual process reviews to automated tests, especially in the ITSM release, change and configuration process areas).

Continuous deployment is a prerequisite for the high deploy rates characterized by DevOps, and is therefore a needed skillset for the modern DevOps practitioner.  It will also be the salvation for a generation of ITSM practitioners.  Read it.

Release It!: Design and Deploy Production-Ready Software

When you watch talks at the Velocity Conference about how organizations like Google, Netflix, Amazon, Twitter, Pinterest and others are engineering code and environments to operate at scale, you’re often hearing many of the ideas that were first promulgated by Mike Nygard in his fantastic book, “Release It!: Design and Deploy Production-Ready Software.”

(In fact, I distinctly remember sitting next to Adrian Cockcroft in one Velocity Conference session, listening to him rattle off the chapter/verse of the Nygard patterns as he recognized them in the talk.  The person giving the presentation should have read the book.)

This is a book that helps span the Development and IT Operations divide, by showing developers and architects how to build applications that can be deployed, managed and survive in even the most hostile production environments.  When you read this book, you’ll see in his patterns and lessons horror stories from your own past.

IT Operations practitioners need to read this book, too, in order to connect the dots of how specific Development decisions lead to bad production outcomes that they’ve  experienced in the past.  And more importantly, it will enable them to go to architecture or development meetings with concrete suggestions on how to avoid them in the future.

When Chris and his team deploy Project Unicorn that performs so well in production in ”The Phoenix Project,” they obviously read Mr. Nygard’s book.

A little story:  I saw Mr. Nygard give an Ignite presentation at the 2010 DevOpsDays in Mountain View, and it was one of the most amazing talks I’ve ever seen.  It was one of the most high-fidelity and intense descriptions of a disastrous application deployment that took weeks to recover from, requiring heroics from IT Operations.  It still remains my genuine hope that the scenes of the Phoenix deployment failure are as disturbing as my experience watching Mr. Nygard’s presentation.  :)

Visible Ops and ITIL Service Support

When Kevin Behr and I started studying high performing IT organizations in 1999, we found that these were the organizations that were simultaneously achieving the “highest IT service levels (e.g., MTTR, MTBF, change success rates, etc.), the earliest integration of information security into the software/service development lifecycle, the best posture of compliance (e.g., fewest number of repeat audit findings), and amazingly, the best IT efficiencies (e.g., server/sysadmin ratios).”

We had studied 11 high performing IT organizations, which included a bank, a stock exchange, a wireless billing service, a domain name service provider, and two IT service providers.

Kevin and I worked together to understand how these organizations made their “good to great” IT transformations, and codified this transformation in the books “The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps” and “Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps”, written by the same author team as “The Phoenix Project”.

A critical part of this journey was made possible by the “ITIL Service Support Book v2”, which no discussion about IT Operations would be complete.  I loved this book when I first read it in 2000, as it catalogued and normalized the key processes that must exist in any high performing IT organization.  But, ITIL remains a descriptive framework, and organizations don’t do descriptive frameworks — they do projects.

Our goal with the Visible Ops series of books was to create a prescriptive, ordered set of steps (projects) to replicate the outcomes we observed in high performers.  (Incidentally, this is what we’re attempting to replicate in the upcoming “DevOps Cookbook.”

Conclusion

Well, I hope this list of resources was useful! Please let me know what you think, or if you have any comments or questions.  Next up will be more about Dr. Goldratt’s Theory of Constraints.

The post Where To Learn More About Concepts In “The Phoenix Project” (Part 1) appeared first on IT Revolution.

Give us feedback!  :)

Q: Amazon.com says the hardcover edition of “The Phoenix Project” book is “Temporarily out of stock.”  How can I get a hardcover edition?

We’re still struggling with getting Amazon.com to order sufficient quanitites to keep up with customer demand.

There are a couple of ways to get the hardcover book:

Option 1:  You can actually order the hardcover book through Amazon, and even get it with free two-day shipping via Amazon Prime.  Here’s how:

Go to the “The Phoenix Project” hardcover edition page on Amazon:

Next, click on the “2 new” under “More Buying Choices”

Next, click on the IT Revolution Press “Add To Cart” button.

There are books available!  Order away!  :)

Option 2:  You can also order the book at 800-CEO-READ, where you can get up to 50% discounts when ordering in quantities of 10+.

Q: How do I order a copy of “The Phoenix Project” outside of the United States?

The Kindle version of “The Phoenix Project” is available through the international Amazon sites.

However, you can only order the hardcover book through the U.S. Amazon.com site.  You can put in international shipping addresses.  Expect about $20 for the shipping charges.

We are investigating international distribution options, most likely for U.K. and Germany first, where the orders have been highest.

Q: How can I purchase “The Phoenix Project” in non-Kindle formats?

Currently, “The Phoenix Project” is only available for the Amazon Kindle platform.

Want it in another digital format?  Please let us know which format you would prefer by emailing support@itrevolution.com.

The post Ordering FAQ for “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win” appeared first on IT Revolution.

(Note that Amazon is currently showing the hardcover book as “out of stock,” even though there are books at the Amazon fulfillment centers.  You can still order the book and get it in two days via Amazon Prime by clicking on “More Buying Choices > 2 New,” underneath the big “Add To Cart” button.  We’re working with Amazon to resolve this issue.  Sigh.)

I wanted to share three of my favorite comments that have come in over the last couple of weeks about the book:

“Every IT and business leader needs to read this book. It reads like a work of fiction, captivates like a mystery and educates like a textbook. There are only a handful of books that have a permanent location in my office bookshelf, and The Phoenix Project is one of them. ” -–Lisa Schwartz, Chief Operating Officer, ITSM Academy

 

“Some books you give to friends, for the joy of sharing a great novel. Some books you recommend to your colleagues and employees, to create common ground. Some books you share with your boss, to plant the seeds of a big idea. The Phoenix Project is all three. It provides a compelling answer to the question, ‘Why DevOps, why now?,’ and should be required reading for anyone whose business depends on IT.” –Jeremiah Shirk, Integration & Infrastructure Manager at Kansas State University, and DevOps enthusiast

 

“The Phoenix Project is practical and useful, emotionally challenging and extremely entertaining. The authors take a topic that normally speaks only to IT process geeks and make it fun, relevant and also terrifying to anyone who has ever watched a project go off the rails. I nearly wept as the protagonists made mistakes I’ve made, and cheered when they saw the light. This is a great read for team members, their stakeholders and their significant others.” – Joe Telafici, Former VP Operations at McAfee Avert Labs

You can read all of the testimonials that we’ve been collecting here.

On behalf of my co-authors, George Spafford and Kevin Behr, thank you to everyone who has helped us this journey!

PS: Enclosed is a picture that my wife, Margueriite, took last week when we got home. We found our first box of books from the printers that arrived, with my my twin boys, Parker and Grant, looking over my shoulder.

And of course, here’s something that I will cherish forever.  I took this screenshot of the Amazon rankings on January 16th, when “The Phoenix Project” was on the same screen as some of my favorite books of all time, such as “Good To Great,” “Lean Startup,” and “5 Dysfunctions Of A Team.”

It’s about time that a book about DevOps and the management of IT can be spoken in the same breath as those books…

Viva DevOps!  We live in amazing times!

The post “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win” Is Now Available! (Belated Announcement) appeared first on IT Revolution.

Watching this talk should be required for any infosec person who wants to see how they can integrate into the daily work of Development and IT Operations.

Of course, they mention what it’s like to be at an organization going through hyper-growth (i.e., the famous “Fail Whale” due to capacity issues). But much to the amusement of all of us, they describe the birth of the Twitter infosec program, triggered by the hacking of the @barackobama account.

That breach resulted in an FTC injunction, requiring Twitter to be secure for the next 15 years. And hence, Twitter infosec was born.

This newly-formed department made some huge strides during the Twitter Hack Week, which occurs once every quarter, where they were able to focus on proactive work. During Hack Weeks, everyone “works on whatever they want, which they then demo to the rest of the company,” similar to what’s done at Facebook. These projects often focus on work that reduces technical debt, automate manual work, improve processes, etc.

They wanted to focus on creating more automation, but anchored in the these framing principles:

• Get the right information to the right people: “writing secure code isn’t a technology issue; it’s about people working with technology.  So how do we communicate better?  We didn’t want to do something that generates a huge report, and then just send it to someone.”“Instead, we need to provide the developer who created the vulnerability exactly the information they need to fix it.”
• Find and fix bugs as quickly as possible
• Don’t repeat your mistakes: “the best predictor of the next bug is the last bug. We found that we were fixing the same bugs over and over. Our goal was to use automation to prevent them from happening again.”
• Analyze from many angles: “we gather data from the source code, from the production environment, and even the user browsers.”
• Let people prove you wrong: “we need Dev to trust us, so they need to be able to tell us about false positives. If we’re wrong, we want them to let us know. And we don’t want to waste their time.”
• Help people help themselves: “most people want to do the right thing, so give them all the context and information they need to fix it and they will. “
• Automate dumb work. “we define ‘dumb work’ as anything that doesn’t require judgement or creativity.”
• Keep it tailored: “building our own tools has allowed us to save time and money, because it allows us to look for the narrow things that we care about.”

Automating Security

Justin Collins spoke about the manual security tasks of reviewing code, penetration testing and handling reports from the external world. They set out to automate all of these activities. He made a fantastic point of the workflow around static code analysis:

Even though the static code analysis step is “automated,” infosec still has to do a lot of waiting: waiting for the scan to complete, get back the big stack of reports, interpret the reports, and then find the person responsible for fixing it. And when the code changes, we have to do it all over again! Even though we’re using ‘automated tools,’ we’re still doing a lot of manual work…. So we wanted to put our robots to work. By doing this, we do less dumb button-pushing tasks, doing more stuff with creativity and judgement.”

Back to that first Hack Week: they built static code analysis into the Jenkins continuous integration process, but there was much more they wanted to do. So they set out to build SADB, the Security Automation Dashboard. The logo is, of course, a sad bee.

SADB takes input from brakeman, phantom gang, csp, threatdeck, roshambo, and the outputs include emails that go to developers and infosec.

Brakeman for static code analysis

Brakeman does static code analysis for Ruby on Rails. The primary author is Justin Collins (@presidentbeef ). It’s had 25 releases in last year. Brakeman runs at any of the following times, but in the ideal, Brakeman runs each time the developer saves code!

This is great. In typical Twitter development, Brakeman runs upon each code commit (git push), and when vulnerabilities are found, SADB will email the developer about how to fix it. Better yet, when the fix is committed, SADB will email them another email, congratulating them for fixing it!

Nice. Having infosec integrated into the development and continuous integration cycle is awesome. Catching it at “file save” time is just awesome.

What has been the results of using Brakeman? By creating fast feedback for developers, showing them how to fix their own issues, clearly they’ve made a difference. (The spikes are usually associated with new releases of Brakeman.)

In the video at 19:06, you can find a demo they gave of Brakeman. At the end of the demo, listen to the applause when the developer fixes the vuln, because they’re not “angry and confused” by the results of static code analysis run. Nice.

Note that the “false positive” button in the vuln notification page is labeled, “Bullshit!” Haha.

By doing this, Twitter infosec is enforcing that code is tested early and often. They aspire that Brakeman be extended in the future to deal with

Phantom Gang for dynamic analysis

Neil Matatall then described Phantom Gang, which does dynamic application security testing (DAST). It complements Brakeman by looking for issues like mixed content, sensitive forms posting over non-HTTPS, old versions of jquery that often pop up when new microsites are created, forms without authenticity token which are prone to forgery, etc.

Phantom Gang is a bunch of node.js processes which emulates Webkit browser sessions, which are spun up as headless browser instances to see what users see. The output of Phantom Gang goes to JIRA, as opposed to directly to developers directly. Why? Often the issues found by Phantom Gang are more difficult to trace to an individual developer.

Interestingly, Phantom Gang was inspired a conversation they had with the Etsy crowd (holy crap, they’re everywhere). They eventually want to extend Phantom Gang so that they can see the effects of any SQL Injection attacks.

Phantom Gang will eventually open-sourced.

CSP

“Twitter is a big fan of CSP. It’s great for enforcing policy and protecting web sites.” Here’s an article on how Twitter uses CSP.

In a typical XSS attack, the attacker injects arbitrary Javascript into a page, which is then executed by an end-user. When a website enables CSP, the browser ignores inline Javascript and only loads external assets from a set of whitelisted sites. Enabling CSP on our site was simply a matter of including the policy in the returned headers under the CSP defined key, ‘X-Content-Security-Policy’.

They’ve configured CSP to disallow Javascript on the page itself, and have Javascript only served from themselves. When there’s a violation of that policy, “it’s almost assuredly a sign that there’s a valid XSS attack underway.” They also use their Big Data capability to look for site spikes, which is often an indicator of XSS attack.

They also use CSP to enforce HTTP Strict Transport Security, which protects against firesheep attacks, which our Dev and Ops people love.

Other great gems from their talk

They also configure TweetDeck configured in a way to monitor the tweet stream, look for specific attack patterns.

Long ago, they played “Ro-Sham-Bo” (paper, scissors, rock) to pick who among the team would do the awful work of importing all the data into SADB. They now have automated all that painful work, but still play ro-sham-bo to assign who reviews all the SADB reports.

They characterized their journey as the following:

• from manual -> automated
• from low visibility -> trending/reports
• from late discovery of issues -> auto notification

• Question: “How long did it take you to get to where you are?”
• Answer: “We went from constant emergency mode to operating more strategically in a couple of years”

Again, watching this talk should be required for any infosec person who wants to see how they can integrate into the daily work of Development and IT Operations. Watch it now.

Video of talk here
Slideshare

The post Here’s How The Amazing Twitter Infosec Team Helps DevOps appeared first on IT Revolution.

My Posts over the past few years…

What Devops Means to Me (2010)

DevOps Driven Demand (2011)

The Five Why’s of Cloud (2011)

The Convergence of DevOps (2012)

DevOps Culture (Part 1) (2012)

DevOps Culture (Part 2) (2012)

Deming to DevOps (Part 1) (2012)

Book Reading List

The New Economics for Industry, Government, Education

Out of the Crisis

The Goal: A Process of Ongoing Improvement

Beyond the Goal: Eliyahu Goldratt Speaks on the Theory of Constraints

Other Great Reads

What is DevOps? (dev2ops)

What Is DevOps? (agilesysadmin)

What Is This DevOps Thing, Anyway? (jedi.be)

The post DevOps Reading List appeared first on IT Revolution.

The first part of my 25 minute talk was about the downward spiral that happens in almost every IT organization, which impacts not only IT Operations, but also Development, Product Management and “the business,” which we’re all a part of.  It starts with fragile applications in production, resulting in the business missing the commitments it makes to the outside world, which then results in urgent, date-driven projects put into the Development queue.  This results in more fragile applications in production, and an ever increasing amount of technical debt, slowing down the organization, starving it of the ability to do productive work.

The second part of the presentation are six of my favorite prescriptive DevOps patterns, derived from the Three Ways, which we’re using to describe the principles that underpin DevOps.  They are:

1. Define work and make it visible.
2. Make environments available early.
3. Wake up developers (fast feedback loops).
4. Embed Dev into Ops (quality at the source).
5. Break things early and often.
6. Reserve 20% of cycles for technical debt reduction.

You can find out more about my upcoming book “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win,” coming out on January 15, and get the whitepaper  ”Top 11 Things You Need To Know About DevOps” here.

The post My PuppetConf Keynote: “Why We Need DevOps Now” (Video) appeared first on IT Revolution.

Etsy is often spoken in the same breath as companies like Netflix, Facebook, Twitter, Amazon, Joyent. These are companies that live and breathe DevOps and showing the rest of the world what performance outcomes are possible by leaving old mental models behind. High deploy rates, amazing stability, reliability and security, and most importantly, a culture that the rest of the world admires.

Community and culture, Rembetsy asserts in the talk, is the foundation of any company. And how does one go about fostering community and encouraging positive culture? You begin by eliminating barriers, getting rid of silos, and encouraging collaboration across the entire company.

For Etsy, it wasn’t always this way. How Etsy got to where it is today is what this presentation is all about. At the time this was presented, Etsy had 350 employees, with 125 engineers, of which 12 are in IT Operations.

Their presentation is structured as a year-by-year retrospective, describing the pains that the organization felt, how they attacked the problem, the outcomes they achieved, and the TODO list that they dragged into the following year.

My notes are below, but you can find the full recorded video recorded by Damon Edward here and their Slideshare link here.

2008: The Year Of Pain And Living In The Sea Of Our Engineering Filth

Rembetsy and McDonnell’s story begins in 2008, back when Etsy only had 30-35 employees, of which half are engineers. The company was doing approximately $87MM in revenue, with 250 servers in two data centers. “This was our Year Of Pain. Deploys took hours. Code didn’t work. There was little or no communication between developers writing the code and the IT operations team responsible for getting it out. Pushes routinely failed, restarting and rolling back changes was a challenge, resulting in HTTP 500 errors across the entire site.”

Deployments would routinely fail, causing “HTTP 500 errors” across the entire site, with no easy way to restart the services or rollback changes. “After a deployment day,” Rembetsy said, “I’d be completely spent and frustrated.”

They were stuck in a very siloed culture, most evident between IT Operations and Development. One of the most egregious examples of this was a tool called “sprouter,” which was specifically designed to prevent engineers from directly doing production database changes. In hindsight, “sprouter” almost guaranteed to create the wrong culture and outcomes by creating an artificial wall between Dev and IT Operations.

One of the first things they did was open up communications between Etsy developers and the customers. They created the http://fix.etsy.com blog to encourage communication and add transparency during outages, not only for the company but also for their customers. They started communicating outage updates including reasons, duration, progress, and expected resolution. This allowed our customers to know why they’re down and for long, and most importantly, “we’re doing this for you.”

On the day before Cyber Monday (the highest volume day for e-commerce sites), Rembetsy kept thinking, “WTF did I get myself into?!” Looking at their project backlog, Rembetsy observed, “We realized that we had to fix our technical debt, and that we couldn’t keep living in a sea of engineering filth,” Rembetsy said. This required focusing on high-improvement projects that mattered, instead of projects with low or no value.

Allow me to interject for a moment, by rephrasing what Rembetsy said, using the terms we’re using in the upcoming book, The Phoenix Project,” because their thinking mirrors our is terms of the necessary prerequisite steps to make meaningful improvements. We believe that the very first thing that an organization must do when embarking on this journey is to do the following:

• Create slack time for important improvement projects
• Keep batch sizes small and the planning horizon short (e.g., weeks, not months)
• Keep prioritizing higher “the system of work” over “doing work”

Back to the talk: They left 2008 with the following promises: 1) Gain support from the top and bottom to change culture, 2) Increase transparency both within the organization and to the public, and 3) Pay back technical debt as soon as possible.

2009: The Year Of Sea Change and Deployinator

They started the year by looking at issues caused by the offices. To encourage collaboration, Etsy put everyone under the same roof in a new office in DUMBO, a borough that was closer and more attractive to the people they were hiring in Brooklyn. “The place where you work must fit with you culture. You can’t have lean, creative, agile in a plain, dull office.”

They also created a “DevTools” team, starting an effort called “Deployinator” to automate their continuous deployment process, which they’ve open sourced. Deployinator enabled quick and safe deployments, by allowing them to create “the smallest number of steps, with the smallest number of people and the smallest amount of ceremony required to get new code running on your servers.”

This was also the year when genuine collaboration started to happen between “people in management and people in the trenches,” ending the era when management would just say, “Go do this” or “Go do that.” People were happy to come to work, and contribute in ways beyond their job description. And interestingly, this was also the year where they eliminated scheduled downtime, because deployments were going so much more smoothly.

They summed up 2009 as the beginning a “DevOps” culture, where the Berlin Wall between Development and IT Operations fell.

They left 2009 with the following action items: 1) Find the parts of your own organization that are causing you the most pain and try to stabilize them; 2) Hire staff that will make a difference; 3) Pick the projects that will make impact; 4) Get it done, just ship it.

(Again, when they say, “get it done, just ship it,” this is what we’re calling “keep batch sizes small and planning horizons short.”)

They left 2009 with $177MM gross merchandise raise (up 103% vs. 2008), 320 million visitors (up 96%), and 9.45 billion page views.

2010: The Year Of Standardizing

This was the year that Etsy brought on Kellan Elliott-McCrea as VP of Engineering and John Allspaw as SVP of IT Operations, who also had worked with CTO Chad Dickerson (now CEO of Etsy). This is when they created the Code As Craft blog, one of the premier software engineering blogs, to share their practices and lessons learned.

This is also when they created the continuous integration and delivery team, creating a fully automated test program, which enabled developers to have the confidence to do high rates of deploys.

They also started to standardize and very deliberately reduce the supported infrastructure and configurations. One decision was to switch everything to PHP and MySQL. This was a philosophical decision, not a technology one: they wanted both Dev and Ops to be able to understand the stack, so that everyone can contribute if they wanted to, as well as enabling everyone to be able to read, rewrite and fix someone else’s code.

They also adopted the mantra, “If it moves, graph it.” Etsy installed screens around the office which showed in real time graphs of what was up and running and what was down, shown in priority order. This way there was rarely questions of what one should be working on or which fire needed to be fought first.

Here’s one of their famous graphite metrics, aggregating metrics in a single-timeline that also shows when all the deployments occur, to enable constant situational awareness of activity inside the organization (e.g., web pushes, search pushes, etc.) and the health of all the services.

In the picture below, note all the vertical lines in the monitoring graphs — each of those vertical lines is a code push, color coded by application/service. This is the famous “vertical line technology that we helped pioneer.” (Haha.)

They currently have over 7000 checks for 700 hosts. “We used to have lots more, but we got rid of unimportant checks that were resulting in unimportant 3am wakeup calls. Stop being woken up at 3am for broken things!”

They created and implemented the “Developer on Call” program, to address the problem of IT Operations asking, “why should I be the only person waking up at 3am?” To create more developer responsibility and accountability, and to ensure that IT Operations had the necessary resources on hand during deployments, each developer rotated to be on call for one week. With the company’s current size, this translates to one week every three years that a developer would have to be on call 24/7.

This furthered the cultural norm that developers take responsibility for rollbacks and fixing forward when deployments go wrong.

They also started continual A/B testing, prototypes (users who opt-in to beta test features will be part of the A/B population that gets served new features), feature flags and ramp-ups, and “Schema Change Thursdays.” This last one is fascinating. Rembetsy says, “We stopped changing schemas whenever we feel like it. Now we batch them all up and do them only once per week.”

The management ideals that started getting formed included:

• Accept failures but don’t lower standards. Failures happen, and it’s best if they’re visible, understood, and used a springboards to greatness.
• Trust but verify.
• Blameless post-mortems
• Welcome one-on-ones
• Career planning
• Happy company = happy community

Their lessons for the year included included:

• Don’t guess at what’s wrong with your infrastructure–graph it.
• Empower developers with responsibility: let them deploy, have them on call, no passwords, etc. ( “Make sure that developers stick around to ensure that their deploy worked.”)
• Clear documented standards and processes are a must, but they aren’t set in stone. They can and should change as your business grows.
• Management should continually check in to make sure employees are happy and satisfied in their work. Rembetsy calls this a critical part of “human management.”

2010 Etsy business stats: Gross merchandise sales: $307MM (up 73%), 534MM visitors (up 66%), 147MM unique visitors (up 58%), and 9.3 billion page views (up 43%).

2011: The Year Of The Reaping: The Death Of Non-Standard Technologies

This is the year where they eradicated non-standard technologies that the entire company couldn’t get entirely behind. This is when Mongo, Scala, CoffeeScript, Python and many more great technologies were taken out of production. (They recommend checking out Ross Snyder’s Surge 2011 talk, “Scaling Etsy: What Went Wrong and What Went Right”).

Incidentally, this is also the year when the toxic tool Sprouter was killed. Rembetsy showed a graph of the number of Sprouter calls over time: “we were able to drive it to zero by opportunistically removing it bit by bit, and finally it died, too.”

This is also the year when many core Etsy technologies were outsourced, joining Deployinator (now being used by Rackspace): statsd, logster and many more. The company decided that all engineers should be contributing back in one of three ways each year:

• Writing blog posts for fix.etsy.com
• Speaking at conferences.
• Open-source something.

(And by the way, this really shows. I’ve been going to many conferences this year, and my observation is that there are always fantastic talks being given by folks from Etsy and Netflix.)

Some of their current initiatives and achievements:

• “IT Operations switched their configuration management from SVN to git,” enabling to have a combined repository with Development. “This has significantly increased our ability by increasing conformity, allowing us to grow the team.”
• “We did the svn to git transition in one weekend, where everybody worked on this to make sure it was successful.” (Nice. Another recurring theme: singular focus among the entire team to get important things done, like Facebook Hack Days)
• Increase signal/noise to noise ratio
• Writing Schemanator to automate schema chgs to reduce risks
• Focus on information security: Nick Galbreath helped create an information security and compliance program.
• They conducted “game days” to test failures before they happen
• More dashboards, with the framework on github
• Improved weekly financial reporting

Here’s some of the 2011 accomplishments:

I noted with interest that despite their use of continuous deployment, and developers having doing routine deployments, they still achieved PCI DSS compliance in six weeks. Rembetsy said, “We haven’t allowed PCI DSS to change the culture of the company. All the separation of duty requirements can still be fulfilled.”

They closed 2011 with $526MM in gross merchandise sales (up 71%), 895MM visitors (up 67%) and 12 billion page views (up 40%).

Their action items list for the next year:

• “Senior management at any technology company should be technology focused”
• “You need to do configuration management, even if you only have two servers”
• “Don’t let PCI compliance change the culture of the company”

2012: Current Challenges

Explosive growth in hiring and in the company has led to some novel and wonderful cultural hacks. They have a game they called “Guess That Admin,” to address the fact that with so many new faces, people don’t get a chance to meet everyone. This is an internal game where the person who recognizes the most new people wins. (Guess That Admin was also during one of their Hack Weeks. The page below uses pictures pulled from their LDAP servers.)

They have events like “Meetsy” (suggested lunch groups to meet people you may not work directly with) and “Eatsy” (where the entire company eats together). They now invite local NYC companies, such as Tumblr, to have mini-conferences onsite at Etsy, creating a community of practice between practitioners.

And because of new engineering challenges, they’re looking at using non-standard technologies, such as Redis.

Current works in progress include:

• Curbing developer boredom by allowing transfers between teams, including between divisions (e.g., engineering to product)
• Developers can now read data from production databases for development work, removing one of the last anomalies between Dev and Production
• Creating a Front End Performance team to minimize site load times

Their list of action items coming out of the year includes:

• Know when not to try things
• Focus on performance early (and by planning in advance that tools will be open sourced, it forces developers to sanitize their code early)
• Allow dynamic allocation of resources
• Never allow size to dictate culture

Rebutting Future Predictions Of Doom

Rembetsy then took some time to analyze and rebut some of the things he’s heard, mostly that predicted that “when Etsy hits 500 people, that’s when everything will fall apart.” He argued articulately that they’ve created a culture that can perpetuate itself, and I certainly agree.

I wish I had some of these insights while I was at Tripwire, where I was CTO at for 13 years. What a wonderful difference it would have made!

Way to go, guys. Amazing talk.

Closing Notes

I mentioned that this post was one of the best presentations I’ve seen on transforming IT. For the record, the other presentation that I’d put up there in this league was given by Kevin Behr in 2003, describing the transformation he helped lead at IP Services. This was actually the basis of what became the Visible Ops Handbook!

I include it here for posterity.

The post One Of The Best DevOps Talks On IT Transformation: “Continuously Deploying Culture” by Rembetsy and McDonnell, Velocity London 2012 appeared first on IT Revolution.

The post DevOps Town Hall Meetings appeared first on IT Revolution.