How We Used Google AdWords For Title/Subtitle Testing For “The Phoenix Project”

In this post, I will describe the six steps we went through to use Google AdWords to test titles/subtitles for “The Phoenix Project” book, how we tested with specific audiences as defined used the UX-style personas for each of the major characters in the book, how we interpreted the AdWords campaign results, and our conclusions and final decision on the book title.

From start to finish, the testing campaign took three days, cost us about $500, and delivered to us data that gave us confidence to make our final title change to the book.

Why We Urgently Needed Title/Subtitle Testing

For those of you who followed the development of “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win,” you may know that we’ve changed the title several times over the past two years.  Even before Summer 2011, the working titles included:

• “When IT Fails: The Novel”
• “When IT Fails: A Business Novel”
• “When IT Fails: A DevOps Novel”

It’s October 23rd, 2012 and we are having a full-day get together of the publishing team, which consisted of Todd Sattersten, Tim Grahl, Hannah Concannon and me. Among many other urgent things, we still had to finalize the cover design within a month.  And yet, here we were, still arguing about the what the book title should be.

If we didn’t decide on it soon, there was no way we’d have book covers ready, which means that we wouldn’t be able to initiate our first hardcover book print run, which would definitely make it impossible to hit our planned January 15th, 2013 book launch.Several folks whose opinions I trust expressed concerns that the book title with the word “fail” just didn’t seem quite right. Others also said that if we wanted our book to be a shorthand way to refer to a growing revolution in IT, the title needed to be more positive.

Out of growing desperation and a need to make a decision, we finally decided to do what any hypothesis-driven practitioner would do: we decided use Google AdWords to do a title/subtitle testing campaign.

I believe this technique is especially relevant to the DevOps community for the following reasons:

• Some of the tenets of DevOps include using short cycle times to enable iterative development, as well as continuous feedback and testing: techniques like this should be used at the very front-end of the feature funnel, to ensure that there is actual market demand before Development even starts working on it, in conjunction with continuous A/B testing to figure out how to maximize the desired business results.
• We discovered some surprising things about our community of DevOps and IT practitioners: specifically, we found more signs of burnout among the Infosec and ITIL practitioners, revealed by how they voted on the candidate titles.I’ll share what these differences are, as well as my theories on why this is the case… and why Infosec and ITIL practitioners need DevOps the most.

Why Use Google AdWords?

Eric Reis of Lean Startup fame writes: “Our goal [in using Google AdWords] is to find out whether customers are interested in your product by offering to give (or even sell) it to them.”

The underpinning theory is that if no one is willing to click on a Google ad for your new book, software service or offering, they will be incredibly unlikely to click on it on Amazon or buy it when they see if on a bookshelf.  You can read his entire tutorial of how to do a Google AdWords campaign for a new potential offering here.

(Historians will undoubtedly note that the theory and practice of methodical testing before committing capital was described by Tim Ferriss in 2007 in his book “Four Hour Work Week”, using both eBay and Google AdWords.  There are many reasons to follow his works — as my good friend William Hertling once said, “When in doubt, copy Tim Ferriss.”)

So here’s how we set up our experiment.

Step 1: Select Two Titles To Test, Using A Placeholder Subtitle

Our goal in the first step was to find a winning title. For now, we would use a placeholder subtitle, so that we only changed one variable at a time.

One of the titles was the incumbent, “When IT Fails.” For the second title, we chose “The Phoenix Project,” which in the novel, was the project name of the most critical company initiative, but was over two years late and over $20 million over budget.

We chose this because almost every one of the early reviewers would mention the project, alongside all the characters. For this reason, we thought it was a promising and suitable candidate title.

We then went into the Google AdWords campaign management management page, and set up two campaigns. Each campaign has two components: the ad text and ad subtext.

• Title Campaign 1: “When IT Fails: Creating Breakthrough Performance”
• Title Campaign 2: “The Phoenix Project: Creating Breakthrough Performance”

We chose “Creating Breakthrough Performance” as the subtitle, because it offered an attractive (if vague) promise that would hopefully entice people to click on the ad.

The sample Google AdWords ads that would get shown to the right of the Google search results are shown below:

When someone clicked on the ad, they would go to a special landing page we created on the IT Revolution site, thanking them for their feedback.

The next step was to target the Google AdWords campaign (i.e., who would Google put the ads in front of?). For that, we needed to define our audiences.

Step 2: Create Personas From The Novel Characters To Create Our Campaign Search Terms

A common theme in the early feedback we got developing the book was that the characters vividly reminded the readers of people they’ve worked with in their past (or in some cases, people they’re currently working with).

There’s a reason for this. Each of the main characters are a synthesis of the types of people we’ve encountered over the past 10+ years. The characters are like user personas in UX design, personifying the archetypal roles, temperaments and personalities that we’ve all seen in IT organizations.

Listed below are the novel characters, as well as the search terms we thought they’d use in Google:

• Bill (VP of IT Operations): “The Goal”
• Chris (VP of Application Development): “agile,” “scrum,” “Death March,” “The Deadline”
• Patty (Director of IT Service Support): “ITIL,” “ITSM,” “service management”
• John (CISO): “it audit,” “pci compliance,” “cobit,” “infosec,” “information security”
• Erik (the “Jonah”/”Yoda” character): “lean,” “lean IT,” “theory of constraints”

Note that we left out any mention of “Visible Ops” or the author names as search terms, because we only wanted to test the influence of the title and avoid the introduction of any bias in the experiment.

We then configured campaigns for each of the personas with the expected search terms, and tested the two titles against them, shown at the right.

Make sense? To find what the Chrises of the world thought, we configured Google to test our ads to anyone who searched for “agile,” “scrum,” and so forth. To find out what the Johns of the world thought, Google would display our ads for people who search for “PCI compliance,” etc.

Shown at the right are all the campaigns that we created.

Work required to date: four hours; Cost to date: $0.00

Having completed this, we were ready to make the ads live.

Step 3: Analyze And Interpret Results From Title Testing

After running the campaign for one entire day, our title testing campaign served nearly 150K impressions, with 82 click-throughs, costing $170 to execute. The average click-through rate was low (less than 0.2%), but everyone assured us that this expected, given the non-specific nature of the ad.

Here’s what the campaign screen looked like, just for your information — don’t bother trying to read it. We put it down just so you can see what a live campaign looks like. Our analysis follows.

Who Clicked Through By Search Term

The top four search terms resulting in the most clicks came from the following:

• “The Goal”: 31 clicks
• “Agile”: 10 clicks
• “ITIL”: 9 clicks
• “IT Audit”: 7 clicks

Our interpretation of this is that people who were searching for the famous book, “The Goal” by Dr. Eliyahu Goldratt were most likely to click on the ad, because it somehow resonated with them. Given how often the people who gave us feedback made that comparison, that was good news.

The next most frequent search terms were were developers, ITIL practitioners and auditors.

Which Title Won?

Across the entire population, “The Phoenix Project” beat “When IT Fails” by about 2:1. However, it wasn’t uniform. Some personas voted very differently.

Here is the graph of the voting results, broken down by persona:

On the one hand, we can see that people who match the Erik and Chris personas decisively favored “The Phoenix Project.” On the other hand, the Patty persona was split, but the John persona was decisively favoring “When IT Fails.”

When I saw these results, I suddenly was reminded of the study that Josh Corman did on information security burnout. The study noted that signs of burnout included cynicism, fatigue and illusions of self-efficacy. As Josh often quips, “Cynicism isinfosec’s core competency.” (Har har, right?)

I think that cynicism shows in how they voted! Although not nearly as decisively, the ITIL practitioners also voted this way, contrasting with Development.

My belief is that anyone who is forced to live with the downstream consequences (e.g., defects, technical debt) of people made upstream in the value chain (i.e., Development), eventually feel powerless and pre-ordained to failure. This is one of the most damaging things we can do to other people as fellow human beings, and of course causes burnout.

If we had more time, I’m sure that if we could have figured out how to better target IT Operations, they would have voted very similarly to Infosec.

Total elapsed time: 1 day; Total cost to date: $170.

Step 4: Create Campaign To Test Subtitles Alongside Our Winning Title

At this point, we felt confident that “The Phoenix Project” was our winning title. The next step was to come up with potential subtitles. There were about eight subtitles that we created, along the following lines:

• “A Novel About IT, DevOps And Breakthrough Business Performance”
• “A Novel About IT, DevOps And Helping The Business Win”
• “A Novel About IT, DevOps And Creating Breakthrough Performance”

Here’s what they looked like in the Google AdWords campaign screen.

Step 5: Analyze Results From Subtitle Testing

We ran this campaign for about three days, which generated ~218K impressions across 45 keywords. We got 86 clicks on nine of those keywords (“the goal,” “ITIL,” “service management,” “IT audit,” “agile,”) with an average of $2.25 per click.

Some surprises before we got to the analysis phase:

• The high cost per click (CPC) for those keywords: I guess the surprise here was that CPC relative to the cost of the Kindle book ($9.95) indicated that using Google AdWords probably wasn’t going to be cost effective for the book.
• Many of the ads that we configured were being turned off by AdWords due to low quality. This was interesting, as it made sure the ineffective subtitles were being winnowed out of the candidate pool. (We also believe that our overly simple landing page might have triggered this. Todd added more words to the landing page to conviced AdWords that we were legit. Were we to do this again, I’d probably seek out the help of an AdWords pro It might be good to consult with an AdWords expert to make sure we weren’t shooting ourselves in the foot, were we to do this again.)
• Again, “The Goal” and ITIL had some of the highest click-through rates (CTR). The top search terms were: the goal (24), itil (17), service management (13), it audit (12), agile (12).

Here were the results of the voting, broken out by personas:

Note how the Infosec persona was the only population that didn’t like “Helping The Business Win!” I mean, it got less than 5% of the votes, where it was a clear winner for the Erik and ITIL practitioners! More indication of cynacism and burnout?  I don’t know, but it’s fascinating to see how differently they voted.

Step 6: Select And Tune The Winning Subtitle

With these results in hand, Todd Sattersten and I got on Skype and studied the vote tallies, confident that we had narrowed the field down to the best candidates.

After some discussion, we decided to go with the “Helping The Business Win” variant.  We did make a one-word change: we changed the “The Business” to “Your Business”. Why? We wanted to eradicate the horrible habit that we IT practitioners often have, calling the organizations that we serve as “the business,” as if we existed completely outside of it.

(Why does that bother me?  We’re a part of the team, too! We’re not a vendor or a service provider! As Bob Lewis writes in his excellent “Keep The Joint Running” newsletters, acting merely like a utility or a service provider instead of a member of the team helping the organization win is the quickest route to being outsourced,)

I wish I had better documented the other reasons why we chose the “Winning” title, however emotional they were. I think I liked it best, because it best conjured up the vision of everyone in IT being part of a team or a tribe, with a sense of shared values and purpose, working together to help the organization win is really the greatest hope I have for everyone in IT.

In our manifesto, we wrote that we want to “help improve the livelihoods of 1 million IT workers by 2017.” That’s not about creating “breakthrough business performance,” that’s about being part of a tribe that is all helping our businesses and organizations win.

So, there you have it: this is how we did the title/subtitle testing for “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.” Having made that decision, we then gave directions to the book cover artist, which shows all the characters in the book positioned around a bubbling volcano of servers, which represents the Phoenix Project.

I’ll write later on the cover design process, which began before we nailed down the title, and how this title/subtitle testing process influenced the final book cover.

Next up

• Next on the backlog, I’ll be writing about more resources and underpinning theories in “The Phoenix Project:” including GAIT, the Gartner RVM model, kanbans, etc.
• How we designed and constructed the novel, and how the Tinderbox software package saved me when mindmaps, outliners, Scrivener and Evernote couldn’t.

Last note: I see that I started this blog article in December 2012. It’s been sitting on the kanban as “Doing” for 3 months, half written for nearly 60 days! Bad Gene! Too much work in process, WIP is truly the silent killer. I obviously need to adjust down the WIP limit in the my Doing lane in Leankit Kanban. 3 months cycle time for 2500 word article? Unacceptable! Live and learn

- About The Authors

Gene Kim

Gene Kim is a Wall Street Journal bestselling author, researcher, and multiple award-winning CTO. He has been studying high-performing technology organizations since 1999 and was the founder and CTO of Tripwire for 13 years. He is the author of six books, The Unicorn Project (2019), and co-author of the Shingo Publication Award winning Accelerate (2018), The DevOps Handbook (2016), and The Phoenix Project (2013). Since 2014, he has been the founder and organizer of DevOps Enterprise Summit, studying the technology transformations of large, complex organizations.

Follow Gene on Social Media

• 
•