Kessel Run is an effort of the US Air Force to build software using principles from DevOps, deploying multiple times per day to support the multiple people who depend upon them. At the 2020 DevOps Enterprise Summit Las Vegas-Virtual, Adam Furtado explored their amazing transformation.
Their story begins in October of 2015 when the US military struck a Doctors Without Borders hospital in Afghanistan, believing it to be an enemy stronghold. Afghan commandos were under fire and the US needed to respond quickly. Unclassified analysis later showed that a number of failures contributed to this devastating outcome: there was no time to fully brief the crew and the aircraft didn’t have the latest data to identify the hospital.
Basically, a failed IT ecosystem caused a AC130 gunship to attack the wrong building.
They failed not due to pilot error, but due to ancient software and stovepipe systems. “What happened here was not some kind of black swan event, it was predictable and it’s going to happen again,” stated Adam Furtado, Chief of Platform for Kessel Run.
Ultimately, for the US Air Force a business outcome failure does not result in a loss of subscribers or revenue, but a loss of lives. But the Kessel Run effort can be used as a case study to improve mission/business outcomes.
Kessel Run was an effort to solve tough business challenges that traditional defense IT wasn’t solving. A small coalition formed to test modern software practices, process and principles. Their focus was on the mission and a disregard for the status quo. They named the group Kessel Run in honor of Han Solo’s famous smuggling route, an homage to their own need to “smuggle” these new ways of working into the Department of Defense (DOD).
At the time, about 10 years ago, walking into the DOD to work was like walking into a time machine, a completely analog environment circa 1974 where multiple collaborative tools like chats and Google Docs aren’t possible. As Adam Furtado says, “You shouldn’t have to go back in time to go to work.”
Eric Schmidt, Executive Chairman at Google, even testified to congress in September of 2020 that “the DOD violates every rule of modern product development.”
These kinds of problems weren’t exclusive to the DOD. According to US Digital Service, 94% of federal IT projects across the entire government were behind schedule or over budget and 40% of them never delivered.
The Kessel Run coalition was watching companies like Adidas and Walmart become software companies. They wanted to transform the US Air Force into a software company that could win wars. So they turned their attention to the key business outcome in front of them: modernizing the Air Force’s Air Operation Center.
There are several physical Air Operation Centers (AOCs) around the world from which the US Air Force strategizes, plans, and executes air campaigns. Due to archaic infrastructure, all this work is done by specific people, in specific buildings, located in specific locations, to access specific data on specific hardware. A brick and mortar approach that’s been in place for decades. The only updates they’ve been able to implement are Microsoft Office updates.
“You might think I’m lying, but a recent search showed 2.8 million Excel and PowerPoint files on one of the servers in one of the locations,” said Adam Furtado.
Gall’s Law states that if you want a complex system to work, build a simpler system first then improve over time. The Kessel Run coalition did just that by starting with a Strangler Pattern to slowly chip away at and iterate this behemoth out of the field at 22 physical locations, each with their own software and hardware.
They started with a specific process: mid-air refueling. This process requires massive coordination to ensure refueling tankers are where they need to be, at the right time, at the right altitude, with the right hardware to refuel the correct aircraft. The process involved several pilots to plan every day using color pucks, an excel macro, and lots of data entry. They had become largely efficient, but only to the point that their brains would allow and they couldn’t react quickly to changes.
So Kessel Run brought in a team to digitize their process, using DevOps principles, extreme programming, and balanced team models. They got their initial minimally viable product out to users in just weeks. This early program created enough efficiency that it kept one aircraft and it’s crew from flying every day, a fuel savings of $214,000 a day.
They kept iterating, and after 30 iterations were able to double the savings, keeping two aircraft and crew on the ground a day. Saving $13 million in fuel a month and they cut the planning crew in half.
This win got them some momentum, but also some roadblocks. The transformation journey is never over.
To keep learning about the Kessel Run journey, you can view the full presentation in the DevOps Enterprise Video Library.
And of course, for the latest stories, attend DevOps Enterprise Summit Virtual – Europe, happening 18-20 May 2021.