Speeding and Scaling the DevOps Enterprise

One hundred percent growth in attendees is an impressive number for 2015’s DevOps Enterprise Summit (1200 attendees in 2015 from 600 in 2014). Yet it still pales in comparison to the growth in delivery frequency reported by several speakers at the conference, from Capital One‘s 6x increase to a reported 25x increase in release delivery at Bank of America.

DevOps Enterprise success stories revolved not only around technical achievements but also business outcomes, demonstrating that the 2015 State of DevOps findings also apply to the enterprise world. Target‘s bottom-up move (read: without boss’s approval) to loosely coupled API-driven services and a modern delivery stack enabled new business capabilities through fast integration with internal (e-commerce fulfillment, cartwheel) and external (Instacart, Pinterest) apps. The consequence was an overall growth of 42% in digital sales.

DevOps can also impact the bottom line through cost reduction, as exemplified by Bank of America‘s 6x production defects reduction in an environment where a single high severity fix can cost up to $75k. CSG International, on the other hand, leveraged test automation to replace legacy components, achieving estimated savings of 9M dollars.

Who Are You Gonna Call? Mainframes!

It wasn’t all about the shiny web and mobile test automation and rapid delivery though.

#mainframelove was in the air after IBM’s Rosalind Radcliffe’s talk on test automation, covering dev/test environment provisioning (tip: zOS on Intel), zUnit testing and how and when to refactor to services which can then be virtualized for testing. Reducing the load on manual testing allowed some of IBM’s clients in the financial sector to improve time to market by up to 40%, according to Rosalind.

Jody Mulkey, CTO at Ticketmaster, kicked off the mainframe and test automation theme during his Monday keynote where he shared his view that DevOps is not fundamentally about enabling technology (some of Ticketmaster’s VMS-based core systems are tested on emulated VAX where Cucumber-style BDD tests run on a regular basis), but about raising empathy, empowerment and being metrics-driven.

DevOps transformation has helped Ticketmaster reduced their MTTR by 90%, according to Jody. That’s 43 minutes less downtime per outage, which equates to savings of 43x money Ticketmaster makes per minute, which is, well… a lot of money.

Scaling = Sharing + Coaching

The organizers’ decision to invite back several speakers from DOES14 to give an update on their DevOps transformation journey has shown a clear evolution pattern. What were one year ago essentially bottom-up, grassroots movements, often starting with more tractable low hanging fruit such as automation and tool adoption, have now shifted their focus to disseminating their learnings and effectively scaling DevOps (first raising awareness, then practices) across teams, departments and even geographies.

Visibility on initial success stories (with the helping hand of information radiators, demos and hand-picked partners in crime) has led to higher management support for DevOps. Consequently, these organizations are now exploring new territory in terms of expanding DevOps culture and knowledge in a non-intrusive way into more hermetic teams and processes.

Note that top-down executive mandates for DevOps transformation as happened at ING in 2013 are the exceptions, not the norm.

So what kind of initiatives are these organizations carrying out in order to scale? Mostly they involve sharing and coaching:

• internal DevOps Days or software engineering conferences (e.g. ING)
• hackathons where participants can practice new skills (e.g. Intel)
• DevOps/IT academies (e.g. HP)
• engineering blogs (e.g. Target – find a list of DOES engineering blogs here)
• communities of practice (e.g. Capital One)

Automation teams’ office hours and explicit mechanisms for collecting feedback from development teams are other activities that take place at Capital One.

Perhaps the most all-encompassing initiative are Target‘s dojos, an internal incubator (physical) environment where co-located teams apply new practices while working on strategic challenges during a full month or more. Coaches are permanently available and the space is also open for anyone to participate and watch demos of the work being produced.

Target also ran an internal mini summit with many of the speakers from DOES14, which raised awareness and participation among mid management.

Speeding Up = End-to-End Ownership – Inertia

Bringing departmentalized people to work together, automate long-standing manual processes, and give up hand-offs that gave them a sense of control is hard work. Speakers frequently referred to the difficulty in bringing teams with heterogeneous skills, backgrounds and micro-cultures up to speed.

“I’m all for progress, it’s change I object to.” —Mark Twain

Interestingly, many speakers encounter more resistance to change among developers and operations people (who in theory are the main beneficiaries in a DevOps culture) than among security and compliance people (who, again theoretically, see their work more challenged in a DevOps culture). Potential reasons for the former include fear of their jobs no longer being needed or a diminished sense of self-worth or place in the organization. Jason Josephy, development lead at Nordstrom, shared his experience of moving from being a lean and DevOps skeptic to a believer.

Ed Bellis, former CISO at Orbitz, explained how (Sec)DevOps is finally bringing together security officers, developers and operations in a delivery team. Codifying and automating security policies and knowledge allows embedding those controls in the delivery pipeline, as opposed to being one more (and often last) obstacle to production. Ed also stressed the need to manage vulnerabilities in production in an automated fashion (for example pulling in zero day feeds). The idea being that there will always be some security debt. Making it visible allows better understanding of priorities and actionable feedback for the delivery teams.

Randy Shoup’s #1 hard-won lesson of the DevOps transition is the need for cross-functional teams with end-to-end ownership of services, from design to deployment but also maintenance and eventually retirement. Teams with a single responsibility in the delivery process gravitate towards a ticket culture (“do what you’re asked”) instead of an ownership culture (“do what is needed”). CSG, Nationwide and Economical Insurance are some examples of organizations moving to cross-functional teams to achieve fast, independent releases of software aligned with the business and customer needs.

“There are only two states: the system owns you OR you own the system.” —Jody Mulkey, Chief Technology Officer, Ticketmaster

Moving away from outsourcing and investing in engineering practices was also frequently mentioned as a cornerstone in increasing quality and ownership of the organizations’ IT systems and intellectual property. Outsourcing does not bring home the benefits of a successful DevOps transformation as people change and lack engagement with the business.

To Re-Org, or Not to Re-Org

Having cross-functional teams does not necessarily mean re-structuring is a must. Many organizations have been burned by consecutive re-orgs, each promising a shiny new world of increased productivity and less waste. Sometimes it’s better to take concrete, doable steps that improve the workflow and bring teams to collaborate on delivering faster and/or with higher quality. Small steps that do not require multi-month long re-organizations and shuffling of responsibilities.

Ralph Loura, CIO at HP, pointed out that a re-org was not plausible for them due to the sheer size of the organization. Instead of teams, they selected strategic internal apps to be developed “the DevOps way.” They also leveraged the power of inclusion (and auditing) that ChatOps provides (essentially an open chat where build, delivery and monitoring related messages are posted in an automated fashion, leading to real-time collaboration between everyone in the ecosystem).

Nevertheless, some talks referenced Conway’s law on how team structure affects the architecture of the system being developed. Loosely coupled, independent teams with a cleared path to production (and pager duties!) are able to deliver faster and forced to build more resiliency in the service(s) they’re responsible for.

The DIY DevOps Manifesto

A manifesto. A roadmap. A charter. The holy grail of DevOps. Call it what you like, at the end of the day what matters is having everyone on the same page on what DevOps and Continuous Delivery mean for the organization and what are the required steps/capabilities to get there in an incremental fashion.

Intel and Sherwin-Williams have also updated their internal maturity models to reflect progress in their journey and raise the bar as the low-hanging fruit gets harder to find/pick.

USAA, for example, took a layered approach to tackle their challenges. Focusing on test data management first, then building a common test automation framework, virtualizing external services and currently working on their build and continuous integration.

CSG invested in a centralized telemetry system that supports multiple views of the data in order to track progress and promote collaboration.

Be Tight or Be Square

Even if the goals are formalized, enterprises at DOES15 are moving away from long-winded process improvement programs in favor of short feedback loops that allow them to continuously assess their progress.

Elisabeth Hendrickson, VP of Engineering, Big Data Suite, Pivotal, stressed how learning organizations thrive on tight and clean feedback loops. Tight means relentlessly removing obstacles that prevent change from going in and clean means information is neither lost (for example direct customer feedback buried in support cases or infrastructure issues falsifying test results) nor polluted by opinions.

“You are either building a learning organization… or you will be losing to someone who is.” —Andrew Clay Shaffer

Speculation builds up during design and coding phases due to leaps of faith on what the customer wants and how the features perform. Continuous integration, exploratory tests, acceptance and performance tests are all forms of feedback that help reduce the risk of release failure before feedback from customer and other stakeholders comes in. Limiting risk depends on continuously executing those validation mechanisms, said Elizabeth.

Michael Bueche mentioned how USAA focuses on what he called “leading indicators of quality” (e.g. test coverage, regression results, code reviews, cycle time) as opposed to “lagging indicators of quality” such as incidents in production.

Want Honest?

One major contributor to the success of DOES15 was the majority of experience reports from practitioners trying to figure out how to adopt DevOps principles and practices in their corporate environment. No silver bullets. No recipes for instant success.

For example, Mirco Hering, DevOps Lead at Accenture, shared his successes and (many) failures trying to apply modern configuration management and testing practices to systems of record (such as Siebel or Mainframe). Hidden source artifacts, lack of APIs, difficulty to automate and provision sensible test environments are a few examples. Leveraging these practices is important because these systems cost so much, Mirco pointed out. You should be able to make the most out of it, including fast(er) delivery and scalability support, and not be locked down to vendors’ working procedures.

Want Awesome?

As always, some talks shined due to the speakers’ capacity to engage the audience. Jason Cox, Director of Systems Engineering at Walt Disney, delivered a “Star Wars”-themed view of how Disney’s organization and its motto “be curious” have helped drive DevOps adoption.

Dr. Steve Spear, author of “The High-Velocity Edge,” broke down the success of high-performing organizations into four capabilities: discover problems, solve problems, spread discovery and lead (i.e. build the previous three capabilities in the organization). The audience listened in awe to his examples, including Toyota’s transformation from a low-quality, high-effort car manufacturer in the late 50s to high-quality, low-effort manufacturer in the late 60s.

“Performance is a proxy for knowledge and skills.” —Dr. Steve Spear, Principal, High-Velocity Edge

Jeffrey Snover, creator of PowerShell, described the battles he had to endure to replace Windows with command line interface at Microsoft and the wider culture change that it entailed.

Mike Bland, strongly involved in Google’s test automation adoption during its exponential growth years, inspired the audience with his emotional speech as he explained why he embarked in the long road to modernize (and reduce cost of) software development and procurement in the federal government. A road full of cultural challenges where all helping hands are welcome.

Want Chuckle?

One of the reports from the DevOps Enterprise Forum was a “How NOT to do DevOps” presentation by members of the committee on organizational design. A humorous approach with a lot of punch!

Charles Nelles sarcastic talk on finding the path of least resistance also triggered quite a few chuckles.

Want Open?

Why would enterprises invest in open source? “Because it’s the right thing to do,” said Tapabrata Pal, justifying Capital One’s first open sourced tool Hygieia, a dashboard to track the health of the delivery pipeline. More open tools are in the pipeline (code name “Analytics Garage”).

Ed Bellis also announced the upcoming open source release of “tattle”, a tool to manage the vulnerabilities in your environment in an automated fashion.

Want Help?

All speakers were asked to mention the areas they needed help with. Common themes included getting executive leadership to understand DevOps; implementing a blameless culture; finding/learning from more case studies of DevOps adoption at large scale; and key metrics for measuring DevOps benefits and productivity.

DevOps Enterprise needs you! Make sure to comment below if you want to help with those or any other areas mentioned during the talks.

A Word from Gene

Gene Kim, organizer, master of ceremonies and #DOES15 twitter warlord, summed up the gist of the conference in the final day:

“Wow, what an amazing three days! Like last year, DevOps Enterprise Summit was when I learned the most.  What made it even more rewarding was that there’s a real community being created here, not only sharing lessons learned, but actively helping each other, with each person bringing deep expertise from different domains.”

When asked about his top reflections from #DOES15, he provided the following list:

“I loved how so many of the people who presented last year on their DevOps transformations have been promoted, which I believe validates the value they created. Many of them have been asked to spread DevOps throughout the organization, which will elevate the state of Dev and Ops practices across their organization, with support from senior leadership. Like you, I loved hearing about the techniques they’re employing: the Dojo at Target, full-time experts holding office hours, internal consultants, and so forth.”

“Just like you, I was utterly blown away by what is possible using mainframes when we use modern development and test practices. I heard so many stories along the lines of ‘my head exploded’ by Rosalind Radcliffe’s presentation on ways that we can bring the mainframe community into our DevOps efforts, and how we can modernize the services that rely upon them. Wonderful!”

“More than ever, I’m excited for DevOps Enterprise Summit 2016!  See you there!”

- About The Authors

Manuel Pais

Manuel Pais is co-author of "Team Topologies: Organizing Business and Technology Teams for Fast Flow". Recognized by TechBeacon as a DevOps thought leader, Manuel is an independent IT organizational consultant and trainer, focused on team interactions, delivery practices and accelerating flow. Manuel is also a LinkedIn instructor on Continuous Delivery.

Follow Manuel on Social Media

• 
•