Industrial DevOps: Bring Agile/DevOps to Cyber-Physical Systems

The following is an excerpt from the book Industrial DevOps: Building Better Systems Faster by Dr. Suzette Johnson and Robin Yeman.

The notion of iterating and accepting “vague” requirements for large-scale, cyber-physical, and often safety-critical systems may seem unrealistic. For many cyber-physical organizations, this has been a stopping point to adopting these new ways of working. However, this is exactly how some of our greatest accomplishments have occurred. Consider the Apollo 11 mission that achieved the first human landing on the moon on July 20, 1969. The level of uncertainty throughout that mission was high, and the need to learn fast was critical. But they iterated, learned, and eventually landed on the moon.

In today’s fast-paced business environment, organizations must adapt to changing market conditions, emerging technologies, and evolving customer needs to remain relevant. Agile and DevOps practices give us the means to do so. Software may have been the point of entry, but it is not the end. Agile/DevOps practices can be adjusted to fit the unique challenges and needs of building cyber-physical systems by shifting our mindsets and changing the way we work. Industrial DevOps shows us the way.

Bringing Agile/DevOps to Cyber-Physical

Since we first landed a man on the moon in 1969, digital engineering technologies have continued to mature with leaps and bounds. The cyber-physical world can take advantage of these digital capabilities to gain flexibility and adaptability even in the physical space, giving companies that create cyber-physical systems the ability to disrupt the market and improve delivery times.

In fact, digital capabilities create the opportunity to shift physical system development into a digital realm using tools such as emulators, simulators, digital modeling, and digital twins. Today, we also have the advantage of computer-integrated manufacturing, 3D printing, and additive materials that reduce costs over traditional methods. These abilities give cyber-physical teams greater flexibility in the design of physical products and the ability to test more frequently.

And let’s not lose sight of changes happening in manufacturing. With the emergence of Industry 5.0 and the smart factory, how work is performed on the factory floor is also changing. The factory itself is now a cyber-physical system used to build cyber-physical systems. According to Industrial Digital Transformation, this is a result of the adoption of the “Internet of Things (IoT), cloud and edge computing, Artificial Intelligence (AI), big data and analytics, blockchain, robotics, drones, 3D printing, Augmented Reality (AR), and Virtual Reality (VR), Robotic Process Automation (RPA) and mobile technologies.” These digital capabilities help reduce the cost of traditional manufacturing methods, enabling development to frequently validate and test multiple design options and create iterative capabilities.

Additional practices such as modular hardware designs and production flexibility, robotics, automation on the factory floor, and other enabling digital capabilities are improving the speed of delivery from the ideation phase through development, production, and operations.

So, with these new technological advances, what’s holding back companies from adopting new ways of working? The challenge may lie in outdated mindsets. The previous constraints of physicality have been reduced, paving the way for a new way of building.

The traditional development of cyber-physical solutions has been conducted through a serial life cycle flow of design, development, and testing (known as waterfall). The stage-gate milestones of this waterfall life cycle focus on completed documentation (and lots of it) versus validated capabilities. This results in creating a lot of documentation but a nonfunctioning system. The net result is slow time to market, lower quality, cost overruns, and solutions that do not fit their intended purpose. Industrial DevOps does not advocate for zero documentation; instead, it advocates for right-sized documentation in concert with ongoing, iterative development and validated capabilities.

These waterfall practices have been used for decades. In the past, the cost of change was much higher, so organizations focused on controlling change to keep costs low. Today, the cost of change has diminished thanks to technological advancements. Today the risk of not changing is the bigger monster. Jeanne W. Ross of the MIT Sloan Center for Information Systems Research stated, “Clearly, the thing that’s transforming is not the technology—it’s the technology transforming you.”

Digital transformation has also increased customer access to information, which has skyrocketed customer expectations in everything from innovation to the speed of delivery through operations. These expectations are prevalent for digital and cyber-physical products, such as smartphones, wearables, smart appliances, medical devices, vehicles, and even weapons systems. 

According to McKinsey, the COVID-19 pandemic further accelerated the digitization of companies and their supply chains by three to four years, with some digitally enabled products accelerated by seven years. It has never been clearer that now is the time for companies that build cyber-physical systems to adopt new ways of working.

Applying the theory, practice, and learnings from Agile and DevOps has the potential to dramatically improve the development and delivery of cyber-physical systems. Companies that solve this problem will increase transparency, reduce cycle time, increase value for money, and innovate faster. Simply, they will build better systems faster, and they will become the ultimate economic and value delivery winners in the marketplace. These practices are especially useful as the systems become increasingly complex with growth in unintended emergent behaviors.

Taking proven principles and practices from Lean, Agile, and DevOps and implementing them at the system level with a common language and mental model is actually a simple idea. That simple idea executed well can cause world-changing ripples in product development.

In 2012, I (Robin) had the opportunity to support fighter jet teams to deliver updates to a legacy cyber-physical system with multiple safety requirements. We had an aggressive but necessary schedule of thirteen months to deliver the updates. Given the schedule constraints, leadership was willing to take a risk on a new approach to development.

I coached an initially reluctant set of teams through an Agile transformation. The teams leveraged a tiered planning approach, decomposing their work by product, working in pairs, developing in timeboxes, holding daily stand-ups, and performing demonstrations of their work. At the close of each time box (sprint), the teams held a retrospective and identified one change they could apply to the next sprint.

The impact the Agile approach had on the system was immense. The system was completed in seven months as opposed to thirteen, with a record-low number of defects in hardware integration. The impact that the Agile approach had on the team and their morale was beyond amazing. Aerospace engineers, who had been building and maintaining aircraft for over thirty years, claimed they had never had more fun during the development process or a greater impact on the system.

As organizations realize the benefits of Industrial DevOps, the opportunity to disrupt the status quo of the entire product life cycle presents itself. For the US Department of Defense (DoD), this is imperative in ensuring the safety and freedom of the United States and its allies. For the space industry, it means getting the edge on space advancements and human discovery. For the broader community, it provides an opportunity to outpace their competition.

According to the National Science Foundation (NSF), “Advances in [cyber-physical systems] will enable capability, adaptability, scalability, resiliency, safety, security, and usability that will expand the horizons of these critical systems. [Cyber-physical system] technologies are transforming the way people interact with engineered systems, just as the Internet has transformed the way people interact with information.”

Despite the clear evidence that Agile and DevOps practices have played a key role in the success of software development organizations, many organizations that build cyber-physical systems have the mistaken idea that their systems are too complex to use Agile or DevOps practices. From experience, we recognize that even the simplest of ideas can be difficult to implement when working against cultural norms. What we need is to look at the problem we have been solving from a different perspective. Henry Ford is credited with saying, “If I had asked people what they wanted, they would have said faster horses.” People did not consider that an entirely new form of transportation could be made available.

Current State of Cyber-Physical Systems

The term cyber-physical system was first used in 2006 by Helen Gill at the US National Science Foundation. According to the NSF, cyber-physical systems “integrate sensing, computation, control and networking into physical objects and infrastructure, connecting them to the Internet and to each other.” Building from this definition, we include systems that are part of private, secure networks and communications infrastructures. These systems, including software, hardware, and manufacturing components, are often complex and costly to build. Many cyber-physical systems have safety and security requirements, making these systems even more challenging to adapt to changing priorities and technologies. According to some, “Security threats have a high possibility of affecting [cyber-physical systems, and they] can be affected by several cyberattacks without providing any indication of failure.”

Cyber-physical systems are everywhere. You see and use these systems as part of your daily activities. They exist across industries in many different forms. Cyber-physical systems can be found in the automotive industry, agriculture and farming, aeronautics and space systems, undersea systems, energy systems, medical and health care, communication devices, smart factories, smart grids, wearable devices, and more.

While there are challenges to the adoption of Agile and DevOps in cyber-physical systems, it is far from impossible. In our next post, we’ll look at some early adopters of Industrial DevOps principles and practices.

- About The Authors

Suzette Johnson

Dr. Suzette Johnson is an award-winning author who has spent most of her career in the aerospace defense industry working for Northrop Grumman Corporation. Suzette was the enterprise Lean/Agile transformation lead. In this role, she launched the Northrop Grumman Agile Community of Practice and the Lean/Agile Center of Excellence. She has supported over a hundred enterprise, government, and DoD transitions to and the maturation of Lean-Agile principles and engineering development practices. She has also trained and coached over four thousand individuals on Lean/Agile principles and practices and delivered more than one hundred presentations on Lean/Agile at conferences both nationally and abroad. Her current role is as Northrop Grumman Fellow and Technical Fellow Emeritus, where she continues to actively drive the adoption of Lean/Agile principles with leadership at the portfolio levels and within cyber-physical solutions, specifically within the space sector. As a mentor, coach, and leader, she launched the Women in Computing, Johns Hopkins University Chapter; the Women in Leadership Development program; the Northrop Grumman Lean-Agile Center of Excellence; and the NDIA ADAPT (Agile Delivery for Agencies, Programs, and Teams) working group. She received a Doctorate of Management at the University of Maryland with a dissertation focused on investigating the impact of leadership styles on software project outcomes in traditional and Agile engineering environments. She am also a Certified Agile Enterprise Coach and Scaled Agile Program Consultant/SPCT

Follow Suzette on Social Media

• 

Robin Yeman

Robin Yeman is an award-winning author who has spent twenty-six years working at Lockheed Martin in various roles leading up to senior technical fellow building large systems including everything from submarines to satellites. She led the Agile community of practice supporting a workforce of 120,000 people. Her initial experience with Lean practices began in the late ’90s. In 2002, she had the opportunity to lead my first Agile program with multiple Scrum teams. After just a couple months of experience, she was hooked and never turned back. She both led and supported Agile transformations for intelligence, federal, and Department of Defense organizations over the next two decades, and each one was more exciting and challenging than the last. In 2012, She had the opportunity to extend our Agile practices into DevOps, which added extensive automation and tightened our feedback loops, providing even larger results. Currently, she is the Carnegie Mellon Space Domain Lead at the Software Engineering Institute at Carnegie Mellon. She is also currently pursuing a PhD in Systems Engineering at Colorado State University, where she is working on my contribution to demonstrate empirical data of the benefits of implementing Agile and DevOps for safety-critical cyber-physical systems.

Follow Robin on Social Media

• 
•