Why We Wrote Flow Engineering

We have both traveled very different professional journeys. Yet from our experiences and our dialogue, three big ideas have emerged: the need for value, clarity, and flow in our work. These ideas felt familiar to both of us, but we’d never been able to articulate them before. Over the course of our careers, we have found that many of the people we’ve encountered have had similar moments where an ah-ha or piece of the puzzle has fallen into place, and they, too, recognized the need for value, clarity, and flow. Here are our stories, which might feel familiar in some way, shape, or form to your own. They provide the backstory to how the practice of Flow Engineering came to be.

Steve

I’ve always had a fascination with complex problems. I grew up among mysterious, discarded technology collected through my father’s work at Bell Telecom. By tinkering with these remnants, I developed confidence in my ability to tackle technical puzzles and get things working, and I became comfortable acting with a healthy degree of uncertainty. But I still sought out frameworks and playbooks to guide my instincts. Sometimes I even read a manual!

I broke into tech with a support role at IBM and worked my way from there into desktop support for a brokerage firm, IT management, infrastructure automation, build and release engineering (on compact discs!), and eventually the cloud.

After attending my first DevOps meetup in 2011, I felt like I had found a community of fixers like me. I was deep into build and release engineering, Agile infrastructure, and automation in a highly project-oriented organization skeptical of Agile methods. There was a massive rift between business and tech. At DevOps meetups, I heard my challenges echoed by everyone working in large organizations:

• missing connection between efforts and value
• lack of clarity on goals, priority, organization, workflow, etc.
• constant interruption, meeting overload, misalignment, and crippling dependencies

I shared experiences of working on complicated and extensive automation that would never be used, attending endless meetings to analyze and explain technical and architectural decisions, and seeing conflict between every department.

I took over the DevOps Toronto meetup in 2014 and started to curate a delicate balance between technical and organizational subjects, gravitating toward an area where I saw lots of challenges and little content: the socio-technical aspect of DevOps. I would hear time and time again from individual contributors that they could do such amazing things if only X would get out of the way. X was always something like management, meetings, Scrum rituals, product folks, or sales folks. What I started to hear was that problems with friction, confusion, obstacles, buy-in, dependencies, and misalignment were repeated themes.

During this time, key learnings began to emerge within the DevOps community. Books like Continuous Delivery, The DevOps Handbook, and Making Work Visible built a foundation of knowledge we now take for granted.

DevOps Days, particularly open space discussion sessions, were my way of connecting with others who focused on the gaps: the “glue work” to bridge silos, automate away friction, and reduce toil. I was focused more on the flow of building than on what was being built. After all, tech changed all the time, but the issues of value, clarity, and flow never seemed to.

In 2014, I attended the first DevOps Enterprise Summit. It seemed the most practical event in the DevOps space—no tech in a vacuum, no theory without application, and focused on delivering business value. It opened my eyes to what was possible when DevOps was applied in complex environments, and there was honest conversation about the hard problems still to solve. Although the specific words rarely came up, there was a great deal of conversation about value, clarity, and flow.

Over the years, I often consulted and advised founders and leaders as they hit familiar challenges and inflection points. They’d ask for help with pipeline automation, platform implementation, infrastructure management, etc. Their actual needs appeared after a quick conversation: confusion and chaos from growth, misalignment and conflict with stakeholders, and waste and dependencies impacting delivery. I didn’t realize this at the time, but value, clarity, and flow were sore spots in every case.

As I stepped into leadership as a CTO, it became increasingly important for me not only to understand the workings of things but to articulate them effectively to my team, to my executive peers, and to that most charitable audience, the board. I had to illustrate the workings, assumptions, and importance of different decisions and how they fit together. I had to communicate with folks who couldn’t care less how the tech worked and only cared what we could do with it—and when.

I felt the pressure from the board, CEO, VP of Sales, and others to ensure my team was operating at a high level of performance and that we could always prove it. That always depended on strong collaboration with every other role in the executive suite and the collaboration of all our reports.

It became increasingly clear as I collided with these challenges that DevOps wasn’t enough to capture the full scope I needed to address. DevOps lacked a language that would resonate with nontechnical stakeholders across the business. Those needs pushed me to a realization that would unlock a new level of performance and change the way I thought about technical leadership forever.

My biggest lesson came when a lucky surge in sales exposed the need to reduce onboarding time to meet demand. When you’re shown a waiting list of fifty enterprise customers and then look at your current onboarding processes, the math quickly reveals a painful truth. I knew it took far too long to get new clients up and running, but I had no idea exactly how long it took. The most painful realization came when I realized that nobody—neither my team nor I—knew why it took so long.

To try and figure out what was causing this long onboarding time, I created a sort of Value Stream Map—a map that laid out the full sequence of activities involved in value delivery to customers—in Google Sheets, using my rough understanding of the onboarding flow. But this map was still full of blanks, guesses, and question marks.

The breakthrough came when I invited my team and other roles to contribute their perspectives. The picture started to form. The numbers started to add up. We could see lead time, cycle time, delays, problem areas, and a clear bottleneck. Beyond the shock of our actual lead time, we were most surprised by how many opportunities we had within our immediate grasp. We could see many easy fixes waiting for simple action.

After a few changes, we were able to reduce onboarding time from six weeks to four days. The effect on my team was an incredible boost in confidence, clarity, and collaboration. But perhaps the most impactful was a sense of control over performance improvement. We went from individually treading water to rowing together in the same boat with a stopwatch to track our progress. We had before and after metrics that we could easily share to show progress and ROI, metrics that sales and the C-suite cared about.

This milestone served as the first lightning-strike moment for me. It also became a business case, highlighting the tangible difference that efficient workflow could make. Because I worked through this method with my team, they learned to apply these practices without me—individually, with each other, and with other roles across the organization. It took the practical act of mapping out the workflow to drive home those key foundational learnings of measuring end-to-end flow and visualization from Continuous Delivery, The DevOps Handbook, and Making Work Visible.

I had finally tripped and fallen into insights that had been there all along. When we had something challenging to tackle, we measured the current state and found the bottleneck. When we had to make a decision, we looked at the data. We applied the playbook across multiple workflows, from our data pipeline to continuous integration and continuous delivery (CI/CD) and product development. And all these insights helped us eliminate waste and spend more time on what mattered most.

The second lightning strike came when I struggled to share this with folks who cared less about the details than the big picture or folks stepping into the context for the first time. A spreadsheet full of numbers isn’t the most user-friendly interface. Eyes glaze over and interest drains like air from a pierced balloon. To communicate with everyone we collaborated with quickly and effectively, we needed something that was clear enough to enable conversations but would also speak for itself if it was shared with others. I began to play with different visualizations of flow to do that job, and this simplified Value Stream Map is what I began to use.

We turned a corner from being an unpredictable, black-box cost center to being a trusted and data-driven technical organization. We started to bridge gaps to sales, product, the rest of the leadership team, and the board with a new sense of clarity and confidence. Flow Engineering is the product of that experience.

Andrew

I think of myself as a full-stack engineer. But most of the world’s problems don’t originate at the software layer or at the hardware layer. Most problems originate at the view and intention layer.

After training in engineering, I spent my twenties and thirties as a Buddhist monk. My focus was on gradually rewiring the way I reacted to and saw the world and on teaching others to do the same. My goal was to reframe my motivation away from chasing my own short-term benefit and toward acting with a good motivation to nourish the social network around me.

My fellow monks and nuns were a community of incredibly good-hearted, dedicated people who had forgone other careers and devoted their lives to building an authentic community of practice and helping people improve their inner peace. We worked in the absence of money, prestige, relationships, and most other comforts. That does a lot to filter for sincerity. I came to assume that those I work with should be driven by a sense of purpose, benevolence, and passion. The daily practices of meditation and chanting instilled in us joy and enormous patience. We worked like athletes, striving to make every word and action count, repeating the same trainings day in and day out.

This is where I developed my passion for building clarity. The payoff in these practices was in the form of insights, new realizations, and understanding. This journey of discovery and learning was a far more valuable reward than money, comfort, or excitement. I regarded the activities of daily living, including making money, as a necessity similar to going to the bathroom: something that needed to be done but not something to waste more time or energy on than necessary.

For fifteen years, I dedicated all of my time and effort to trying to develop the spiritual communities I was part of. But optimizing for hard work is different from optimizing for impact and effectiveness, and I felt far less effective than I wanted to be. It was clear that just working hard and with good intention did not ensure that the work would be effective and sustainable. I had retrained myself to value the organization and our students but had overextended and devalued my long-term well-being.

In 2014, I left the life of a monk, in part to learn more about how effective organizations were run. I wanted to understand how to make wise use of people’s precious and limited time and energy and how modern management techniques could be harnessed to create a stable, sustainable flow of benefit for others.

A lifelong technologist, I reentered the workforce as a developer with a focus on Salesforce.com, a technology I had implemented for one meditation center. I joined a consulting firm and was quickly tasked with a job no one else wanted to do: the Salesforce deployments. At that time, little information and few tools existed to automate such processes. I gradually realized there must be a way to automate away the toil and free up more time for the creative, interesting parts of the work.

This realization inspired me to investigate how techniques like version control, continuous integration, and automated testing could be brought to the Salesforce platform. I knew if we could improve these processes, we could reduce the confusion and inefficiency that was so common for me and my fellow developers.

In 2016, a colleague pointed me to the State of DevOps Reports, which I studied meticulously. I started watching videos by Jez Humble to understand technical concepts. But as I listened, I noticed that many of his insights were on the nature of human interactions around software development. Again and again, he redirected credit and blame away from individual engineers and toward the system they were working in.

I started to realize that there was a community within software development focused on human and social transformation. I read Continuous Delivery by Jez Humble and then followed Jez’s influence into topics like product management and his book Lean Enterprise. That’s where I first became aware of Value Stream Mapping and began to lead Value Stream Mapping exercises and design Value Stream Management systems. Everywhere I looked, it kept coming up as a theme.

In 2018, I attended my first DevOps Enterprise Summit and had my mind blown hearing Dr. Christina Maslach explain the origins of burnout, Dr. Steve Spear explain the ways in which Toyota revolutionized the manufacturing industry, and the CEO of Compuware talk about engineering their turnaround. The content irreversibly expanded my sense of the role that IT could play in making organizations more effective. I felt like I had found my people in the DevOps community.

At the 2020 DevOps Enterprise Summit in Las Vegas, I attended a Lean Coffee on Value Stream Mapping that Steve was hosting. Participants were curious about how to move from talking about Value Stream Mapping to actually applying it in their organizations. The topics of the day included how to get started, what resources could guide the practice, and where it could be applied. Those questions are no less relevant today. In an increasingly distributed, cross-functional, collaborative world, shared clarity is rarer than ever.

Soon after the event, Steve and I reconnected. We both agreed that tech leaders needed further guidance on this topic and that we could team up to close the gap. We started meeting for three-to-four-hour jam sessions on weekends. We both shared a preference for digital collaboration tools like Mural or Miro, which provided a spacious canvas for our conversations to emerge and insights to coalesce.

Our dialogue revealed an enormous amount of overlap in our interests. It was fantastic to find a collaborator with a common passion. By mapping out our ideas, experiences, and understandings, we built a shared environment that we could easily discuss, revisit, and refine over time.

Early on, we both agreed on the importance of shared clarity and how much clarity was lost in typical team interactions. It quickly became a recurring theme in our work together. Lack of clarity was accompanied by lack of alignment around business value. These deficits combined to create a lack of individual and collective flow. Value, clarity, and flow came up day after day.

Steve had already started organizing his thoughts into the Flow Engineering scaffold. Our conversations deepened and expanded the relevance of these practices into an open, adaptive, collaborative series of practices that can help align teams around value and move them from complexity to clarity, from friction to flow. We bring these practices to you in this book.

Who This Book Is For

We’ve written this book for our peers working in technology in large enterprises. However, we’ve applied and seen the techniques described in this book in contexts far beyond our immediate frame of reference. It’s easy to adapt and tailor to varied situations. It’s flexible enough to help teams of any skill level, and it’s robust enough to be used for ambitious process improvements or day-to-day problem-solving.

We’re specialists in digital product development and delivery, but we’ve seen the need for value, clarity, and flow throughout many roles, industry verticals, levels of hierarchy, and stages of growth and scale.

This book is written for professionals familiar with the basics of Lean and Agile but unclear how to start, restart, teach, or make measurable progress with confidence. This is a book for curious problem-solvers struggling to help their teams or organizations see the big picture. This is for those grappling with complicated frameworks and operating models and wondering, “How?”

We’re aiming to help mid-level leadership in complicated enterprises, but we’ve applied and seen this material used in a broad range of roles and contexts like product management, Agile coaching, technical leadership, architecture, marketing, sales, project management, design, customer success, and more.

Why does it work in all these cases? It’s a lightweight, flexible practice of building a path from where you are to where you want to go. Once you’ve grasped the concepts and techniques, you could find yourself applying Flow Engineering in contexts we’ve never imagined. If you’re looking for a clear, approachable, step-by-step system for building value, clarity, and flow across your organization, this book is written for you.

Order your copy of Flow Engineering: From Value Stream Mapping to Effective Action here.

- About The Authors

Steve Pereira

Steve Pereira has spent over two decades improving the flow of work across organizations. He’s worked through tech support, IT management, build and release engineering, and as a founding CTO for enterprise SaaS. He serves as lead consultant for Visible Value Stream Consulting, as a board advisor to the Value Stream Management Consortium, Chair of the OASIS Value Stream Management Interoperability technical committee, and co-founder of the Flow Collective to bring flow-focused professionals together. Since 2017, he has been developing and facilitating Flow Engineering to make flow improvement in large organizations accessible, collaborative, and actionable.

Follow Steve on Social Media

• 
• 

Andrew Davis

Andrew is Chief Product Officer at AutoRABIT, focused on the next generation of DevSecOps on the Salesforce platform. He is also the author of the leading book on the Salesforce development lifecycle, Mastering Salesforce DevOps. He was formerly Senior Director of Methodology and Training at Copado.

Follow Andrew on Social Media

• 
•