DevOps Is Not a Hobby but a New Avenue to Revenue

The following is an excerpt from a presentation by Oliver Cantor, Sanjeev Jain, and John Scott of Verizon titled “DevOps Is Not a Hobby but a New Avenue to Revenue.”

You can watch the video of the presentation, which was originally delivered at the 2018 DevOps Enterprise Summit in London.

Sanjeev Jain

Let me start with a story. About three years back I walked into an Executive Leadership Team (ELT) meeting. That meeting is made up of a CEO for the EMEA region, a CFO, a CMO, head of sales, head of operations, head of engineering, and me as a CIO.

I was the chief supplier in that meeting, until that day when the head of sales says, “Oh, here comes my most important customer,” and I thought, “Who’s he talking to?” That’s when the penny dropped on me, he said, “And Verizon, when we’re developing networks, products, services, we’re selling it to the CIOs and CTOs of the world.”

Now as the CIO for the international region of a Fortune 32 company globally that invests $17 billion every year with 150,000 plus employees and has offices in all parts of the world, if our products and services are not good enough for you to consume internally, there is something wrong.

I went from being a chief supplier to the chief customer of the same products and services. Speed or the time to market wasn’t my only responsibility anymore. The quality, the efficiency, and a customer’s delight in our products were the important ingredients.

For me that was a perfect definition of DevOps, speed, efficiency, quality. That was also the day when we embraced DevOps into the life cycles of everything we did. We embraced DevOps into how we transformed the products and services. We embraced DevOps into our customer engagement measuring the NPS and the customer satisfaction score. We embraced DevOps into empowering our employees with the digitizing the experience. And we embraced DevOps into optimizing our operations.

We had to go through a whole new technology stack and scale it to every product and every software release we did.

• We had to change our organization.
• We had to change the kind of talent we were nurturing. We had to create a culture where IT or the technology organization wasn’t in the back office supporting business — it was our business.
• We had to change the organization’s culture where our product development team came and talked to us about the vision rather than the user stories.
• We had to change the culture of our organization where product development teams came to us and talked about the urgency, the speed to the market. In return, we gave them a competitive advantage by delivering high-quality software, fast.

Today we’d like to share with you about one of those use cases, via Oliver Cantor who’s the director of product marketing. He can explain how he had a vision about transforming the network.

Oliver Cantor

I’m excited to share with you something that I would say has been very successful, and a real market drive for us.

What do we do? Verizon traditionally builds global networks, manages themselves and multiple layers, IP networks, MPLS networks, optical networks. These are big vast static ecosystems globally that move, traditionally pretty slowly. They’re fixed, the engineering probably hasn’t changed radically in about 10 or 15 years. People buy big chunks of bandwidth. They think of us as expensive, slow moving, taking a long time to deliver anything.

With all the pressures that are facing enterprises — the fourth Industrial Revolution, climate change, Millennials, social media, 5G, digital transformation, the blockchain, you name it — every type of business is under a lot of pressure. They all need better answers from IT but they also need better answers from the network.

About four years ago we had a vision.

We decided that we needed to make our networks agile. This is unaware of what was going on in IT at the time, unaware of agile and the whole movement of producing software quicker.

We traditionally bought things from vendors, the Cisco’s of the world, very slowly implemented them, put equipment out in customer sites, customer premise equipment (CPE,) data sensors, big chunky bandwidth around the world, submarine cables, etc. This was no longer going to be sustainable. We needed to change the network, transform them to be agile, to be at the speed of software.

What did we build? Our vision was to have an entire ecosystem of software components, things like:

• Traditional routing and software defined routing and security and optimization, we wanted in standard software blocks now.
• We wanted to deploy them on traditional equipment, on new standards-based white boxes and gray boxes, server-based equipment, put them in our data centers and ultimately put them in the cloud.
• We also needed middleware or orchestration layers to manage all of this automatically, zero touch provisioning,.

You name it, it went on and on. The ask was absolutely massive for IT.

Let me give you an example.

A very big bank wanted to run a new application in their branches. Branch real estate is really dying as a concept for banks. They don’t know what to do to reinvigorate the branch experience. They wanted to run an app that would stream video and do an ‘Ask the Expert’ kind of session.

Normally that would take us months to deliver a router, a firewall, an optimizer, lots of expensive heavy bandwidth that we would implement — normally MPLS, a lot of truck roll, a lot of people going to site, etc. It would take months and months and months.

We decided we could roll out one box, a white box. We could do it on hundreds of sites. When it was configured by whoever was on site, i.e., plugged in, it would automatically over Wifi, find home and build itself and self-configure through the orchestration systems. Then they would be up and running in minutes rather than days, and weeks, and months.

Now we’ve saved them months of implementation time, which is all customer experience and business that they’re getting, so it’s been a massive success.

All this, this massive IT task was started in 2015 when we were asked in May if could we deliver a minimum viable product by August. We presumed they meant the following year, but no, they meant in three months’ time. We’d never done anything at such scale and so fast. We wouldn’t even have the requirements at that time, but we did.

Ultimately, we couldn’t have done this without our IT partners. I think most importantly, we needed much more than just a technology answer, we needed a whole cultural change from IT, which is what we got.

Sanjeev Jain

I mean he really didn’t give us an easy challenge. For years the way we’ve consumed the network with routers and CPEs, etc. he just wanted to put everything as a software.

We had a whole suite of ingredients that let us transform ourselves. But right at the heart of everything we did was DevOps. It was John Scott who really took Oliver’s idea as vision and converted it into reality and started a DevOps journey.

John Scott

The question was, how are we going to build up muscle?

We went for an in-your-face learning star called the Verizon Dojo, and this is the Dojo in Basking Ridge.

This is how we’re going to build quality software to keep pace with market demand. This is the setup. It’s light, it’s modern, it’s breezy, sort of place you want to feel inspired to come and work. As you can see, there’s plenty of areas for breakout sessions to aid collaboration. There’s plenty of technology there. There are jam boards, there’s click shares, there are iPads. It’s a sort of place we want you to come, get creative, think big, and go for those blue sky solutions

This is where teams come with their business partners and they spend six weeks learning about engineering with their Dojo coach. Sounds a little bit like training but, here’s the key, it’s training on the job, bringing their real work, real business problems for real business outcomes.

Let’s delve a little deeper into the Dojo.

How does it all begin?

It starts with a charter, gives teams a goal, a sense of purpose, a raison d’être.

What happens?

They spend six weeks iterating very, very quickly towards their various MVPs learning all the time about advanced technologies from their Dojo coach. This is the important bit. It’s all about gaining knowledge, doing stuff, failing fast, understanding your context and constraints, and then iterating, all within a very short cycle.

How many Dojos do we have?

We have five, three in the US and two in India. We’ve now gone upwards of 50 teams that have actually gone through the Dojo journey and we have a big backlog of teams waiting to start their journey.

We’ve upskilled over 500 engineers now and what we’re seeing is on average a 30% increase in speed of delivery for those teams that have gone through the Dojo. But it’s not just the flat metrics, but it’s also the feedback. People who once talked about being in a silo, lost. Now actually feel like they’re part of the team, they’re excited, they’re energized, and that’s what we’re seeing. If I can take these people from being skeptics to believers and evangelists, that is how we’re going to build the community from the grassroots up.

A couple of patterns emerged coming out of the Dojo

One is sometimes teams cannot physically be co-located for six weeks together. We’re hoping later on this year we’re going to bring you Dojo in a box which will be a mixture of that physical Dojo experience as well as virtual.

We also realized we needed to pivot the organization and the way it was structured. Let’s move away from silos. Let’s think about being mission focused, stable teams. We had a squads and tribes model. For Oliver, we created an SDN tribe which created two squads underneath, a BSS squad. These are the guys that are looking after, quoting through to the cache. Whereas the VNS guys, these are the more people that are interested in actually getting the product into the customers’ hands. We had to be thinking about product based, thinking along the whole stack towards business outcomes.

How did our engineers find this? This is the exciting bit. They’re actually full of energy, they’re interested in the product, “how are the sales doing? What else can we do next?” I don’t just want it to be about delivering software. We don’t just want it to go, “Oh, it’s come in my in-tray and it’s gone out my out-tray and I’m going home now and I don’t really care.” It’s about the excitement around that product, and that’s the key.

The other thing which we needed to provide was chapters so we could have like-minded people all talking the same language so our engineers could go across squads, otherwise, we’ve just created more silos. We started with development, testing, analysis, but latterly we’ve gone more granular so now we’re just talking about machine learning, artificial intelligence, monitoring, databases.

Then we asked, ‘what else could we do for our teams?’ We’ve given them an immersive learning experience. They’re product focused. They’re excited. They’re energized. We need to give them a blueprint, we need to start thinking about practices over tools, give them some guide rails with a supporting technology radar.

I introduce to you the Verizon delivery pipeline.

The key here is that it all has to do with implementation and integration.

Let’s start by taking a look again at Oliver. He’s got another aha moment, and says “Guys, we need to go and do this.”

We’re like, “Oh god, here we go…”

What happens next? We created a funnel process to get all those ideas into our issue tracking system, into JIRA, and we somehow need to get it from there all the way over to production in a better, faster, safer way.

I’m a developer now. I don’t really want to go and spend time in JIRA and context switching. I want to be adding value, which is in my IDE, so in Eclipse for instance. We need to implement and integrate JIRA into the IDE, and so on.

If you take a look at our commit stage up in the slide, you then start moving into source code management, so there my IDE is integrated into my source code management.

Then what’s going to happen? We then string up maybe unit testing, maybe performance testing, maybe some level of security scans in QA.

It’s all to do with implementation and integration. We have to remove those manual wait times and laborious manual processes.

We’ve broken this down into a number of stages to keep it simple for folks. We talk about enablers, those are the smaller boxes and stages. We have a commit stage. We have various testing stages, all the way through into release stage, into production.

Now, this was great for solving what Oliver wanted for one product, but unfortunately, in our space, we’ve got like 50 more Olivers and we had to scale this. Let’s take a look at a couple of the ideas around how we can help you scale this. Let’s bring on the gamification.

We’ve got something called Verizon DevOps Days which we’ve recently re-badged Verizon Tech Days because we’re embracing a wide-ranging set of technologies across the whole of the organization. These are internal conferences. They are driven by the community, for the community. They’re the guys that set the agenda. Let’s take a look at some of the metrics, 7,000 attendees across five global events, a great mix of internal as well as external speakers, 400,000 Twitter impressions. [Slide does days]

Then there is the DevOps Cup.

This is something we’re particularly proud and is pretty unique to Verizon. This is all about rewarding teams for doing the right things, growing their capabilities and contributing back to the community

It all started in VES by a few of us who were trying to make engineering a little bit more fun. We started small and then we slowly built it up and up. For instance, in 2016 we had a security first focus. Then we started increasing more events. We then started doing flash builds, etc. The key here is you start small. Make sure you align to your corporate goals and build that core muscle. Please do celebrate successes. We really need to celebrate because change is tough and organizations just don’t want to change, they are resistant.

Where are we at in 2018?

350 applications, thousands of developers across the globe, and now eight lines of business which is all lines of business in Verizon — we’re all encompassing. Now the question becomes, how do we score this transformation? We’ve now got everyone involved, excited, wanting to win. Clearly, I’m not going to be able to manually score this anymore. Google Sheets, it’s one of my friends, but it’s not going to save me this time. We need to ensure we’ve got telemetry across the whole of that pipeline, all those different enabler points.

What we found is a lot of teams were measuring a lot of things, but not necessarily the right things. One of my key phrases is no more vanity metrics. So we partnered with CapitalOne on their Hygieia product to standardize and streamline our data collection. This has allowed us to automate, baseline, and track over time our data collection and also our scoring for the cup. What’s great about Hygieia is it’s very lightweight to onboard for teams, and secondly, it also interacts with tools that teams use on a daily basis so like the Jenkins and the JIRAs. We’ve made it a no-brainer for teams to jump on board. You need to make it easy for people.

But what makes up the cup?

• Engineering. This is underpinned by the pipeline.
• Technology. Are you a modern app? Are you able to refactor your app? Do you have a well-architected framework?
• Community. Are you building that community? What are you delivering back? Are you doing tech bytes which are small elevator pitches? Are you doing megabytes which are the longer talks? Are you running events? What are you doing back for the community? Are you sitting with your sales teams, talking about how digital IT is and they really understand how you can help them? What’s great about this model? Sanjeev know this is, it’s very simple, E, T, and C.

Finally, what happens at the end of the cup? The top-scoring teams go through to the final, and in the end, they are judged by the internal Verizon DevOps gurus. Next, the demo and present to external thought leaders.

Here are some of the things that they are being judged on.

• Business Resiliency. Stability. Are you guys just waiting for a production to go wrong and then you’re going to get phoned up, fire-drill, middle of the night, “Oh my god, oh my god, we need to fix production, we need to fix production? As we heard yesterday we’re just sticky plastering it and moving on because everything’s changing so quickly.” Or are we practicing productive business resiliency? Are you guys trying to run chaos scenarios and test, or even in production? If you are, we’re going to reward you in the cup, and you will do well.
• Security. Are you just firing off scans and forgetting about the feedback? “Oh, I’ve run the scan, tick, it’s compliance.” Or is there a good feedback loop? Are you actually failing so if you do code commit, you run a security scan, you have some critical, you have some highs and actually it breaks the build? If you do, that’s pretty sophisticated in my situation. So we will reward you for the cup.

The exciting thing is it’s really giving our developers a sense of purpose, it’s made learning fun, and that manifests itself in the metrics. We’re seeing a 50% increase in production deployments.

Let’s talk about applications and their maturity level. Can you go from a crawl to a walk, to a run? Legacy should not be seen as a bad word. To keep the momentum up how are we doing that? Smaller rewards, more recognition rather than just a big bang finale at the end of the year.

And to be honest, when John and their team ran the first cup in 2016, teams competing against each other to score points and optimize their operations, that was the most excitement I’ve seen.

Overall, however, I like to think of it as this is the plumbing. We are promoting best practices across all our teams at scale regardless of their maturity.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
•