The following is an excerpt from a presentation by John Rzeszotarski, SVP, Director of Continuous Delivery and Feedback at KeyBank, titled “Breaking Bad Leadership: The Anti Heroes of DevOps.”
You can watch the video of the presentation, which was originally delivered at the 2018 DevOps Enterprise Summit in Las Vegas.
My name’s John Rzeszotarski, and I work in the infrastructure group for KeyBank.
A little bit about KeyBank:
- We’re located in the Midwest, the Northwest, and the upper Northeast.
- We have 20,000 employees.
- We have $137 billion in assets, which puts us at about the 15th largest bank in the United States.
- We have anything from mobile banking, online banking, deposits, credit card, equipment finance, we do real estate capital. We have tons and tons and tons of products.
But it didn’t start that way.
Our roots go back over 200 years. With that complexity came a lot of acquisitions, a lot of new services, and a lot of new products. It also came with a lot of new systems that we had to build, a lot of new people and processes, and lots and lots of layered security.
For an operations resource, it can feel like a ticking time bomb seeing all of this glued together, and then as an operations manager, your team starts sending you really interesting texts in the middle of the nights.
On top of all that, we have to deal with something else that’s happening in the industry, and that’s all of these great startups building all these amazing things and releasing every day. We have to do the same exact thing, only we have to deal with all that complexity.
And this is the complexity that we have to deal with:
This is just an example of the infrastructure systems that sit on the top, the monitoring systems that sit underneath, and then the teams on the bottom that utilize those. This is one of those optical illusions, you have to stare at it and make your eyes fuzzy, and you can actually see the silos within the organization.
It actually was most notable in 2014— we suffered a very significant outage and we said, “Look, we have to do some things differently because we just can’t understand and unravel this complexity.”
And this really all culminated in us going to into DevOps Enterprise Summit 2015. We came out of that extremely inspired, and saying, “Alright, let’s put together a pitch, but first, we need to make sure we understand our problems.”
And I think one thing that I’ve heard a lot at the conference, is that it’s all very contextual, it’s all different based off what your organization is and what they’re doing.
Luckily, there were some major themes that were very similar.
- We knew we had a siloed organization, we knew we had way too many manual processes.
- We had tons of tribal knowledge, and that was creating bottlenecks and all kinds of other problems.
- We weren’t talking amongst other teams.
- And most importantly we weren’t actually trying to fix these issues, we were almost just admiring them.
Identifying Focus Areas
We went out and we picked a specific use case that we wanted to go work on. It was actually perfect timing because we were in the process of rewriting our online banking and mobile banking solution, and we wanted to partner with them.
A lot of people will say to go pick low hanging fruit to get started with, but if you want to make a meaningful and impactful transformation in your organization, pick a use case.
These are our flagship applications. We chose these because they are the most important applications we have in the bank today, and we said we’re going to automate and re-platform the infrastructure components, and we’re going to move fast with them.
We can’t move fast, however, unless we test fast.
One thing we also noted, was yes we were building this new platform for our new application to run on, we also had to do the same thing for the testing platforms, and that was something that we had to manage and take care of from the infrastructure team’s perspective.
Some of those test results that very impactful, so we more than doubled our test coverage.
Likewise, what we used to call test automation took 20 hours, and we would only run it at the end of a cycle, and we never found any defects. We were running way more tests in less than 12 minutes and we were finding anywhere from 10 to 20 to 30 defects during the middle of that development phase, which meant that we were finding it way earlier in the process, and that really allowed us to move fast.
We delivered the application early because we were also in the process of acquiring another bank called First Niagara. This meant that we had to onboard 500,000 new customers. The one problem we had was with this one enrollment page, and we screwed up. It was complex, it was convoluted, we just didn’t build it quite the right way.
But this is where DevOps sold itself.
We did 10 production releases in the first four days. We did them in the middle of the day, we did them during the highest volume of traffic KeyBank has ever seen, and we did that all with zero defects and zero customer disruptions.
That’s something I’m extremely excited about.
ATTEND THE DEVOPS ENTERPRISE SUMMIT
Continuing the Journey
This really sold DevOps for us from a leadership team’s perspective, and a lot of teams came knocking.
I’m really proud to say that Docker, Kubernetes, and a lot of the patterns and practices power all of our customer-facing applications today. To say a 200-year-old bank that’s 15th biggest in the country is doing those things, is something that I’m really proud of as well.
One other big success story we had was our account origination’s application.
This is the front door for any new customer, and it continually changes and gets tweaked. But it’s also the front door for fraud attacks. This is something that we always need to be able to change, and change quickly.
This team is extremely appreciative because what used to take them four hours on a weekend now takes less than five minutes, which is a big success story for them as well.
“What would you say you do here?”
When I was trying to put this together, I was thinking about sharing some of the design patterns we’re doing in the infrastructure space, or about how we’re using Kubernetes in all kinds of different ways.
But Gene had a different question for me, he said, “What would you say you do here?” And so I wanted to answer this matter of factly.
My job and responsibility are to run our code and release management organizations. I run container orchestration. I do that not just for software development, but also for infrastructure development.
This has gone on to include things like monitoring and feedback so we can capture all those analytics and create those feedback loops, build new capabilities with cloud engineering, and automate provisioning process as part of cloud engineering. And then build IT service development so we can remove the bottlenecks, build IT self service capabilities, and move fast.
We’re a platform team— we federate out to the rest of the organization, and we’re the evangelists as part of that. But our job is solely to try to continuously improve the rails that the bank is running on and building software on.
But that was not what Gene was really asking about. What Gene was really asking about was, what my your role in the company is.
Make the bank better
I honestly define our role as making the bank better in every possible way, and that doesn’t mean where I sit within the organization. I might sit in the infrastructure space, but my job is to federate out and identify areas all throughout the bank, and every single vertical within the bank.
We do that in a couple of different ways.
The first thing we do is to go out and target specific candidates — we’ll be very outward reaching.
We do that because we know some areas benefit greatly following some of these principles, practices, technologies, and capabilities.
The second thing we do is what we call utilize modern tech to ‘mark to market’ expense.
We’re measured by an efficiency ration. Normally, when we want to do new things, we want costs to be cheaper or at least, keep them flat. We’re at an interesting time, and a lot of people are talking about it, but open source can drive that ‘mark to market,’ and get fair market value for what you’re doing.
And that’s the thing, in an enterprise, you would normally get strapped by what a vendor was forcing you to do. Now, however, we can actually say, but based on subscription licensing, based on the power of open source, that barrier of entry is extremely low to start utilizing these. And the abstraction and design patterns that you can bake into your solutions allow you to easily flip flop these different solutions and platforms.
Lastly, we always are striving to identify bottlenecks and inefficiencies.
I’ll give you one example here.
One of my peers runs the help desk, and she was reviewing the list of top calls into the health desk. The top three calls were all related to password resets. And we said, “Hey you know what, we could probably throw together a quick chatbot to see if we can automate that specific use case.” We got it up and running over a weekend, and it wasn’t much longer after that that we were actually able to roll that out and reduce the counts.
Then that took off and spread like wildfire until it was like, “Well, we have all these other great use cases, and everyone’s throwing out all these different ideas. We just created a bit of a problem.” We now had to solve how are we going to capture the right intent or the right capability that they’re striving to do.
Once again we put on our smart hats and said, “Let’s look at natural language processing and some of the A.I. capabilities that you can call with the cloud services.” It’s so easy to just go in and start utilizing services like that, but this ended up as another big success story for us.
ATTEND THE DEVOPS ENTERPRISE SUMMIT
The Power of Persuasion
The more I thought about it, there’s another big piece that’s missing, and that’s really about the power of persuasion.
The other thing I think we do really well at the bank is to persuade other leaders. Our CTO calls this ‘the departments of no,’ and they are the things that we have to change and we have to break through.
At the end of the day, it is about perception, because what got a leader in that position of leadership is what they trust, and what they know that works for them. In this case, you have to be empathetic in order for you to respond back and use your power of persuasion to change their minds.
Operations leadership tendencies
Ultimately, I think this relates to several different tendencies that are representative of things like ego, poor planning, holding people accountable, or enforcing and crossing standards.
I thought what might be useful here is if I take these and I wrap them up into some of the personas that we deal with and talk about DevOps case studies that I’ve used in the past to help that power of persuasion.
The first one I call ‘The General’
I think we’ve heard the theme of ‘command and control’ a lot. And that’s Frederick Winslow Taylor, he wrote the Scientific Principles of Management, which is really about how the power is concentrated at the higher levels of the bureaucratic structure, and that’s where decisions are made.
But if I put myself in an empathetic position and say, “Look, I’ve lots of times tried to put a decision out to a group and they can’t get an answer.” And that’s where command and control worked for me in some instances. I need to remember that, but also change that story a little bit, and say rather than ‘command and control,’ it’s ‘empower and delegate.’
And there’s no better use case in my honest opinion, especially for resources struggling with this, than Stormin Norman Schwarzkopf, who wrote the book It Doesn’t Take a Hero.
In there he has the ‘14 Rules of Leadership,’ one of them says, “Don’t tell them how to do the job, simply allocate resources, set standards, and the results will exceed your expectations.” Norman was huge at driving autonomous teams into the military, and it’s an amazing use case and story for you to help the power of persuasion with some of the leaders that you’re working with.
The Boy Scout
Another persona is one I like to call the boy scout, and these are folks that follow standards on a metaphysical level (that’s not really possible,) but common sayings would be, “I know it’s not right, but it’s the policy,” or, “What exactly is your charter Mr. Rezeszotarski?” Anytime you get that one you’re like, “Uh oh this is not going to be a good conversation.”
If I put that empathetic position back out, and I say, “Okay this is the persona that wants to continue to utilize policies to secure the bank, to secure the organization, and it’s the right thing to do.”
While that’s exactly right, but I think another way to flip that persona is Dave Farley.
And looking at the fact that I’m going to continually safely, and test our controls through experimentation.
The reason that’s important is that Dave Farley says, “Look, humans are always going to make errors, like to err is to human.” We know that we’re not going to understand the complexity, and the only way for us to drive that accurate success is through experimentation. Lots of times I just point people to videos of Dave Farley because I think it’s a great persuasive story in order for our leaders to see the importance of experimentation.
The last persona is the Skeptic.
Common sayings might be, “We would have to engage and enlist several departments,” or a persona that says, “That’s not something we have funding, time, or resources for.”
This is the guy that actually knows all the requirements that drive your change— he’s your subject matter expert. He’s almost your product manager for change because he knows exactly what you have to do and what needs to change. He just needs to understand that disruption is coming, one way or the other.
There is a lot to say about the importance of disruption, and how it drives into every organization because every organization’s a technology organization.
One other big part about disruption that I think is neat is if you look at things like improvement, and some might even classify innovation, it could be defined as ‘doing the same things better,’ but disruption is really doing new things that replace the old way.
Honestly, I think that’s where DevOps actually sits, and it’s also using DevOps to create more disruption in your own verticals.
That’s the one thing I think people really need to understand, we’re not improving the process, we’re just doing a new thing that’s making everything else go away.
I never did answer the question, ‘What’s different at Key?’
Safely empowering our employees. This starts with our CIO Amy Brady.
She very much embodies embracing change agents, driving continuous learning, and continuous improvement, and understands the fact that we need new models. With paradigm shifts, you cannot follow those same old models. Whether that’s the same old networking playbook from an operations perspective or the same storage playbook that companies have followed for years. There’s a new model that we all have to get on board with.
Amy Brady wraps all of that in making sure that we all understand that we are the CEOs of our own careers, which really gives us that enablement and that ability for us to go out and operate with autonomous teams.
Another really cool thing that I think we do is transformational architecture.
John Willis and I were talking about this one night, about the role and responsibility of enterprise architecture, and how in many organizations it’s really just a governance mechanism. It’s around are you following our tech standards, are you not following them.
I think what’s different at Key is we don’t approach it that way at all. We don’t have enterprise architecture, we have transformational architecture because we understand that we have to keep up with the times. We have to embrace these new things because they’re only going to help us.
When you do really cool things with a lot of your systems, you get asked to collaborate on a lot of really cool things. This slide was actually presented at Google Next, and we’re collaborating with them on the alpha implementation for Google Kubernetes engine.
It’s something my team is really excited about because what’s coming in technology is coming fast and it’s great.
The last thing, and it’s a no-brainer, but it’s people.
Our people are willing to experiment, they’re empathetic, and I think we’re ‘half cup full’ people, and that really is a testament to celebrating small victories. And if you want to drive culture change within your organization, you have to celebrate every small victory that’s out there.
That could be that one guy finally checked in a piece of code into source control, that’s a victory you need to go out and celebrate.
We have to get comfortable with being uncomfortable— I think that’s so important. The most important thing about that is you have to be courageous.
I love this quote by Meg Cabot: “The brave may not live forever, but the cautious do not live at all.”
If courage is part of the solution, then what I want to know from the rest of the community, and the rest of the conference is, what are you afraid of? Because I think that’s what we ultimately get excited about.