Project to Product: How Value Stream Networks Will Transform IT & Business

The following is an excerpt from a presentation by Mik Kersten, CEO of Tasktop, titled “Project to Product: How Value Stream Networks Will Transform IT & Business.”

You can watch the video of the presentation, which was originally delivered at the 2018 DevOps Enterprise Summit in Las Vegas.

For my entire career, I’ve been on this journey to help transform how software is built.

I spent the first ten years of that journey, as a developer, writing a ton of code. I thought the problem was actually with the code, and with the way that the code was disconnected from the values stream. That’s how I ended up writing over a million lines of open source java code that’s still in use today in frameworks such as aspect-oriented programming that you might know from Spring, tools like Eclipse, my own project that influences how developer tools work, and so on.

What I realized was that if we remove impediments from developers and connect them to the users that they’re supporting, we get a sense of focus, flow, and even joy. I actually got my PHD in software engineering on exactly that topic.

When my supervisor and I realized this, we decided we wanted to try to scale it. We wanted to try to make this work for the very large scales, and then we thought where better to look for those scales than enterprise IT. We started a company Tasktop to do just that.

I learned some very different lessons about what it’s like to work in Enterprise IT and at that those kind of scales than what it was like in open source.

I actually noticed that a hundredfold productivity difference between how quickly open source developers, now imagine a community of hundreds of open source contributors, could create value compared to what I was seeing in these very large scale enterprise IT organizations.

I realized the problem is actually much deeper. We’re no longer trying to improve the lives of 10,000,000 developers, we now have to think about all 40,000,000 IT specialists. And so, I can quantify this past decade of my life not with lines of code but with traveling over a million air miles (which I can assure you is nowhere near as fun as rewarding as writing a million lines of code.) It did teach me, however, a whole lot of valuable lessons by allowing me to meet with hundreds of IT leaders who were struggling with this problem – struggling with this weird and broken layer that we’ve put between technology and the business.

To try to bridge those gaps, I created this thing called the Flow framework that I will share with you today. The Flow framework is what’s featured in Project to Product, and the goal is to give us a common language between the things we’ve learned over the past 10 years of agile and DevOps, and connect the way that the business looks at technology and surviving and thriving in this age of digital disruption.

My journey started at Xerox Parc

It’s 1999, I just got my undergrad degree from the University of British Columbia in Computer Science and Anthropology. I got this opportunity to work at Xerox Parc which was an amazing. This is the computer science lab where they invented things like the graphical user interface, little things like the mouse, and our objective in programming was pioneered there as well.

I was having the time of my life. I was getting to work on these super cool new programming tools that were supposed to bring business requirements closer to the developers and expressing them in the code. I was also working about 80 hour weeks trying to do this because we had a thriving user community and I loved what I was doing.

And after some time, the RSI started. I got this really bad tendonitis and then that progressed until some nerve issues in my arms. I tried mousing with my other hand and so on, but at one point my boss told me that he’d seen several careers end this way, so he suggested some paid time off. This was the absolute last thing I wanted to hear. I was having the time of my life building these tools and coding, and I was far from burnout, but my body was not holding up. At that point, I decided to figure out why this was happening; why was it that I couldn’t keep coding these longer hours? I was having so much fun doing it.

I started analyzing my own work, and the most surprising thing to me was that it wasn’t the coding that was giving me RSI, it was actually all this thrashing I was doing with all these windows, they had stack traces, they had tickets, they had instant messaging, there was e-mails (because we were tracking user stories in e-mails at that point,) etc.

We had done all our continuous integration and continuous delivery automation, so it wasn’t even that, it was just all the communication in term of what we were supposed to be doing, and how misaligned our software architecture was to that flow of value-stream artifacts.

I thought ‘okay, if it was misalignment, well, we can make a software tool to solve that,’ so I did this experiment of creating Mylyn, and it actually fixed my RSI because it bridged this gap between what I was working on and the structure of the code. Other people seemed to have this problem as well, so I started seeing millions of monthly downloads, and that was kinda the foundation for my PHD thesis.

I realized there was something really fundamental around this weird disconnect between the software architecture, and the value stream. That was my first epiphany that’s featured in the book.

These fragmented value streams kill personal productivity. In my case, that actually caused me physical bodily injury because I just kept wanting to code, but in other cases, you may end up with burnout, people who are unmotivated, or who just can’t get work done the way they want to.

Fast forward to 2007

This is what my phone looked like— I thought it was beautiful.

It turned out that, Nokia had started that agile transformation, they realized that they were about to get disrupted.

They knew about the large screen of the iPhone a year ahead of time, and so they realized that they had better get at software. Clearly, this was a company that was incredible at manufacturing physical devices, but the screen was about to get so much bigger and involve so much more software that they knew software delivery would be their bottleneck, and they realized this in time.

They started adopting agile practices, they started hiring more developers, they did all that things that companies are doing today. From the outside, it appeared like they were doing it the right way. I was invited to speak at the agile conference in 2009 about how to connect the agile and open source, and I was amazed because every room I went into, there was a consultant on stage saying how Nokia was a poster child for Enterprise Agile.

But then I started working with Nokia, and I realized the truth was very different from the way they were being described. The way the business had been looking at the agile transformation was to see how many teams were agile, how many teams were trained on scrum?

Which meant the measurement of the success for their agile transformation was this proxy metric. If you’ve seen Jeff Bezos’ letter to share holds in 2016, he talks about how proxy metrics, are metrics of activities, not metrics of a result.

This test Nokia was doing for agility only looked at whether they had optimized a very narrow segment of the value stream, development. When Kevin Fisher mention that when Nationwide looked at their Internet value stream, only two and a half percent of the time was spent actually in development.

Imagine, Nokia could have hired twenty times the amount of developers and it would not have helped this problem. And as a result of the business not realizing this, Nokia actually completely lost the mobile market, that it had created. This was really my second epiphany, that the silos, and measuring proxy metrics, was not going to be enough to make a transformation successful.

In fact, these are the very things that destroy and derail transformations as we saw with Nokia.

Now it’s 2016— we should have learned all these lessons

But now I’m at a bank, which is one of the top 25 banks by revenue, and they’re in their third transformation, which means that the first two failed in very similar ways to Nokia.

The third one was not just an agile transformation though, it was an agile and DevOps transformation. They were also taking the approach of automation as well, so there was some fundamental positive differences. I thought, ‘okay, this must have a bit of a chance of success,’

This was also such an important transformation that there was a billion dollar budget for its success. Everyone was pushing for making this thing to work, everyone wanted this thing to succeed. Then I realized that it was going to fail. I noticed that there was another very big silo, IT was doing the transformation basically within IT.

There were chief digital officers (and those kinds of people) who were brought in of course, and they were creating these amazing dashboards for the bank, but because it was separated, those dashboards would never work in any of the regions that they were planned for. The software architecture was never going to support it, and the silo that was separating the business side, and IT was completely getting in the way.

The transformation was being managed to of course cost, because it was a project management transformation, and so the costs were reduced at the end of the two years. In addition, after interviewing the people involved, everyone said the same thing, ‘Our ability to deliver value was also significantly reduced,’ which is, of course, because it was just managed to account for cost alone.

This layer that was put between the technology side, and the business side, completely got in the way for this transformation to succeed. And that led to my third epiphany, project management and cost centers are completely the wrong model if you want to become a software innovator. In fact, they can completely derail these transformations and we have to replace them with something better. Moreover, we have to do that quickly because other companies already have.

Below are the stock price change between 2006 and 2016 for some retailers

Now, the retailer that has the big bar, that 1.910% stock price increase, that’s Amazon, which has completely aligned its business strategy to its software products, to its software architecture with microservices, to its organizational structures. They’ve completely out-innovated everyone else in the process. But this is not just happening in retail.

We’ve come to this point where some organizations have done this. They do not have this project management layer that’s assuming a high degree of certainty of what will happen for the next two years of the market, they’ve become software innovators.

Now, the change has become so rapid between those who’ve managed software delivery at scale, and the other organizations, that at today’s churn rate, half of the SNP 500 will be replaced in the next ten years. This is happening, and it’s happening quickly.

has this ever happened before?

I think for all our careers, we’ve been used to all consistently crazy change, javascript frameworks coming out every two weeks and so on. But have we ever seen a rate of change like this before?

Gene Kim introduced me to the work of an economic historian by the name of Carlota Perez, who has studied technological revolutions. In her work, she looks at the last five technological revolutions, starting with the Industrial Revolution, the Age of Steam and Railways, Age of Steel and Heavy Engineering, Age of Oil and Mass Production, and where we are today: the Age of Software and Digital.

One really fascinating thing that she discovered is that all these revolutions have two phases, there’s an installation period, and there’s a deployment period, and between them, there’s a turning point.

In the installation period, there’s a big frenzy because some new means of production becomes very cheap. For example, you might have mass production.

This was the period in Detroit over a hundred years ago, where there were 300 car startups which was fueled by financial capital. Another example of this is recently there was a conference on Fintech in Las Vegas, where there were over 2,000 fintech startups.

These companies are mastering the new means of production. What’s interesting is that this causes a tremendous amount of creative destruction, which has happened in every one of these technological revolutions. Until some masters of the new means of production emerge.

We can now call those the fangs of Facebook, Amazon, Netflix, Google, you can include Microsoft, Alibaba, and so on, these companies have mastered the new means of production. They’re amassing great amounts of wealth along the way. I think the key thing right now is to help all the other companies in the world economy move into this wealth generation that happens when we have the dissemination of this means of production.

how do our companies survive this turning point?

What do we learn from all those tech startups? What do learn from the tech giants? What are they doing differently than Enterprise IT organizations (who are still responsible for the majority of the world economy) are doing today?

And how do we have our companies survive this mass extinction event that happens in the turning point, because we are currently in the middle of that turning point?

If we look at those revolutions, each of them has brought with it a new managerial discipline. There have been new financial systems discovered, things like Factory systems, Subcontracting, then Taylorism, and then Fordism.

What’s so fascinating and from conversations from Carlota Perez this became clear, is that Taylorism is actually where we learn how to do really interesting amazing projects. The Hoover Dam was created with Gantt charts, which is an amazing thing. But you could not create an economy of mass production that way, where mass production actually manages lean methods, and products, and value streams.

What I’ve noticed happening to these companies is that a lot of them were founded and established before the start of the Age of Software in the ’70’s. And they were still using management techniques from two ages ago to run their businesses.

What I wanted to know was how did these mass production companies survive the last turning point? I began to look at a company that we’ve worked with very closely, BMW, who are clearly one of the masters of mass production, but still made it through that last turning point. What did they do? How was their business connected to production, and could we learn anything from that?

I had this amazing opportunity to visit the BMW Leipzig Plant, and the thing that was so amazing to me about this plant is not only the architectural elegance, but the complexity of this plant is tremendous.

There’s a new car delivered off the end of the production line every 70 seconds, and those cars are delivered in the same sequence they were ordered, it’s called ‘just in sequence delivery.’ They have an ecosystem of 12,000 suppliers putting in parts into these cars, and of course, many suppliers providing the different robots and parts for the production line as well.

As you can see, complexity isn’t an excuse for Enterprise IT, this is somewhere near that same scale of complexity, but no one there is talking about project management. They’re living the lean principles that are listed from lean thinking.

Those principles are:

• Precisely specify the value by product
• Identify the value stream for each product
• Make value flow without interruptions
• Let the customer pull value from the producer

And so in this plant, everybody knows where the bottleneck is, the CEO knows what the limiting factor of the plant is.

Meanwhile, in all those meetings I did traveling around these past few years, I asked many, many times “What is the bottleneck of your software production” and neither IT executives or business leaders would give me a straight answer. I’d just get vague answers or a blank stares.

What is so amazing about BMW and their ability to architect everything around these value streams, you can actually see the value stream architecture from space.

This over here is the one in two series building, this is a very long, high-automation line, because it’s delivering those cars every 70 seconds. No manufacturing step can take more than 70 seconds.

The cars weave in there, and this is extensible, they can add new manufacturing steps by extending those building. BMW did not know how much demand there would be for their electric series, I3 or I8, this might it a very configurable line. So, everything is configurable as these value streams, and the business understands the product lines and the value streams.

Let’s compare this to what we’re doing today with Enterprise IT.

In car production, we’ve got…

• Everything is beautifully integrated car production lines, not these disconnected tool chains.
• Everything is managed as products, not as projects.
• Everything is architected around flow, not technology layers, front-ends, back-ends, and so on.
• Everything is architected around these product lines.
• Everything is optimized end-to-end, not optimized in silos.
• Everything of course is managed by business result and not these proxy metrics of one aspect of the silo.

Fundamentally I think the problem is that we’ve not agreed upon what production with productivity is in software delivery. We don’t have a common definition.

If we look at business flow of the BMW plant, it’s about quality that delivers sheer driving pleasure.

Interestingly, those cars are designed in yearly cycles, even though they’re delivered every 70 seconds, so there’s a difference between what they’re doing what we’re doing, but the creative and manufacturing processes are decoupled, they’re in separate buildings, and of course, all of this flows across a linear line. Line stops, everything stops. That’s a little bit different than what we see.

In IT and technology and software, we delight our customers not with releases, because those should happen continuously now, but with new features. They’re pulling new features, they’re pulling new value. Those things are designed sometimes on a weekly or daily basis, and the creative and manufacturing process are one, so that’s a pretty fundamental difference. The flow is also not linear, it’s not a batched process of delivering a feature. It’s a flow across a complex value stream network that includes all the specialists, designers, developers, support staff, and operations involved.

The flow framework comes to answer this core question, ‘What flows in software delivery?’ We need a common definition of productivity and of flow, and it’s these four units.

1. Features: that’s new business value that’s pulled by the customer.
2. Defects: quality improvements also pulled by the customer.
3. Risks: so the reduction of risks, security, availability, compliance, and so on pulled by people like risk officers.
4. Debts: technical debt improvements, and these are pulled by the architects, by the developers, and so on.

And these four flow items are mutually exclusive and comprehensively exhaustive, which means they represent all work that’s being done. You can have these very specific taxonomies for example in the scale agile framework. All of those types of work items mat into these four flow items.

I’ll give you an example 

Imagine your last big push to market. Of course, with a big push to market, it’s your ability to deliver great features to your market and that’s probably going to define the success of that product. As you do that, without push to features, there’s a cost because it’s a zero-sum game between those four flow items which means that you take on additional debt and risks.

And the result? Quality of course goes down as you do that, and defect work goes up, and up, and up, to the point where you can easily get into this death spiral where the defect work keeps rising, this is where we can actually get the burnout. Developers want to be delivering features, they want to see the impact of their work on your customers, but of course, they don’t get to do that, and that’s where we can actually measure.

And this is exactly the situation that Nokia got itself into. And just as a case in point, look at this tweet by John Cutler.

In 2015, a reference feature took 15 to 30 days. In 2018, the company’s probably now bigger, 150 to 300 days. Of course, the technologists understand why this is happening, they know they need to invest into technical debt reduction, the problem is with the organization, or business not understanding that.

The whole goal of the Flow framework is to provide a common language

This idea of the flow framework is that you’ve have this flow distribution, and you’re measuring for flow metrics.

• How many of those flow items do you deliver for a sprint, a release, a year and so on?
• What’s the efficiency of that delivery because we can measure that in the value stream network?
• How long did it take for a feature or a fix to get to market?
• Flow load, we know that putting too much load on a value stream causes things like burnout and actually decreases productivity, so we’ve got a measure for that.

The key thing is you do this for each value stream, you don’t just do this for operations, you don’t just do this for one feature team, you don’t do this just for a scrum team — you do this for a value stream which tends to be bigger than feature teams, and you correlate that to business results that have to be specified for that value stream.

For example, those business results could be revenue. If it’s an internal product such as an internal developer platform or API that can be an adoption metric.

You measure costs, but you measure costs for the value streams across all the different specialists involved in building up the value stream.

You measure happiness, but you measure happiness with measurements like employee net promoter score for all the people working on that value stream. Then you have an early warning indicator that if you put too much load on one set of teams, you could cause them to burn out and cause developers to start leaving.

Lastly, with project management, things can go on forever, but if you see business results dropping while you’re adding more features, maybe it’s time for that thing to die.

Then there are things like Flow Efficiency. For example, we measure end-to-end flow time, rather than just looking at development cycle time. We really need to take the customer’s perspective of flow, which has to do with the end-to-end activity ever since that feature or defect fix was submitted.

Then we can create these dashboards, the key thing being, those dashboards are all for each value stream, so you’ve got this level of visibility. This won’t tell you if you’re bringing the right features to market, but it’ll show you when you’ve got a bottleneck, or when you’ve got one of your time fees, such as too many dependencies between a value stream slowing one of the value streams down.

Now the question becomes, how can we do this?

How can we go from silos and proxy metrics to this connected value stream network that we can measure, that we can optimize, that we can manage, and that we can see?

The idea here is that you take all these different tools, and create a tool network out of them. You stop thinking about individual tools, you create this abstract model that you can actually measure.

We call this the artifact model that sits on top of your tool network. You can then measure flows across that network, and you can measure those flows and map them into your product value streams, and measure the flows in those product value streams.

You can go from this waterfall world of projects, where again, you’re assuming that you know exactly what you should be delivering for years on end, to this flow orientation but measure the results of that flow orientation.

You can go from this world where everything has to be specified up front, to creating a project development budget, that allows for the reallocation of budgets between project value streams on a much shorter time frame than a year. Then you can adapt to your market for wherever you are getting those business results.

And of course, this key thing is that you stop bringing people to the work — you bring work to the people. You allow those teams to form stable value streams that are bigger than just feature teams.

Feature teams are a great start, but you give the entire value stream, the ability to have autonomy, mastery, and purpose, and you allow them to set the flow distribution.

Those teams, the product value streams will know, when they should reduce more tech debt for example, to reach a north star, where as where they should run faster on feature delivery.

My closing advice

On the business side, ensure that you’re defining these product portfolios and these product value streams. Ensure that the metrics are being tracked. Empower the delivery team themselves to allocate flow distribution, they’re the ones who know when they need to reduce tech debt, you just need to listen when they need to be given that chance.

This is why we need a common language, this is why the leaders of the tech giants know when to listen to their delivery teams.

Sometimes, you’ll actually need to do things differently to hit a north star, for example, you might need to do a major risk reduction, a major investment, and things like GDPR and you need to accept that that’s gonna take away from the flow of, say, feature delivery.

To technologists, create your value stream network by connecting all those tools and abstract over that tool network. Define your value stream architecture, this is as important as your software architecture. Then use those flow metrics to identify bottlenecks. This will be your early warning signal, or you’ll look at where to extract the platform because everyone’s got dependencies on this one internal component or team.

The idea is that we go from projects and cost centers, to these product value streams, from silos and proxy metrics, to flow metrics, and business results / fragmented value streams, to this integrated value stream network.

- About The Authors

IT Revolution

Trusted by technology leaders worldwide. Since publishing The Phoenix Project in 2013, and launching DevOps Enterprise Summit in 2014, we’ve been assembling guidance from industry experts and top practitioners.

Follow IT Revolution on Social Media

• 
• 
•