Big Data and Lean Thinking: What Makes the Whole?

You are here

Big Data and Lean Thinking: What Makes the Whole?

Posted August 18, 2016 in Business Agility & Software Engineering Excellence

What might be streamlined and automated today may be entirely replaced tomorrow by innovative, disruptive human creativity. As the strategist Arie de Geus observes, “The ability to learn faster than your competitors may be the only sustainable competitive advantage.” This is the essence of Lean Thinking and how it can transform the behavior and culture of an enterprise, preparing it to compete in a highly disruptive future.

Machine learning can help people gain new insights and make informed decisions, but people can’t always turn the running of a process entirely over to machines, especially within a complex and dynamic system. While a machine may surface hidden signatures that no human would think to look for, we strongly believe that humans must be able to see that result and, at a minimum, hypothesize the mechanics that allow for a relationship between those new Xs and the Ys. This requires special insight and creativity.

If the analytics engine is a black box to the people responsible for the work, imposed on them by experts, it can become a blunt instrument of management command and control. Automated monitoring and sophisticated analytics can help the team to monitor the state of a process, and instantly sense and respond to deviations, but the team must understand their process to understand the meaning of the deviations so they can solve the right problems and make the right decisions to achieve the desired outcomes.

Big data has already demonstrated many successes, and experts assert that cognitive computing systems can actually make the context behind decision making “computable,” acting as a proxy for human intuition. It is that convergence — human creativity supported by relevant information — that offers the greatest potential.

From a Lean perspective, then, how and when can we make context computable in order to help those who are closest to the problem? To address this question, let’s briefly examine the fundamental problem-solving journey embodied in Lean Thinking. In his article “Purpose, Process, People,” Jim Womack observes:

... business purpose always has these two aspects — what you need to do better to satisfy your customers and what you need to do better to survive and prosper as a business. Then, with a simple statement of business purpose in hand, it’s time to assess the process that provides the value the customer is seeking. Brilliant processes addressing business purpose don’t just happen. They are created by teams led by some responsible person. And they are operated on a continuing basis by larger teams led by value stream managers.

The further elaboration of Womack’s Purpose, Process, and People in Figure 1illustrates the inner feedback looping and learning of continuous improvement, where the people responsible for the process are able to guide its improvement. When an organization is able to incorporate this mindset into their management systems, behavior, and culture, we can say they have become an adaptive learning organization.


Figure 1 — The Six P Model.

A common challenge with traditional management systems, however, is that they attempt to control a complex, dynamic process from the top down and outside in, focusing primarily on outcomes. Trying to reward and punish those doing the work so as to achieve the desired outcomes can lead to tampering and gaming, where individuals and teams hit their quantitative targets but cause negative consequences in other parts of the organization. W. Edwards Deming argued for constancy of purpose and the abandonment of quantitative quotas, numerical goals, and the popular Management by Objectives approach. In their place, he emphasized skilled leadership.

Deeper understanding of the in-process measures can help teams understand the true drivers of performance, regardless of where the responsibilities and account­abilities reside, so they can continuously fine-tune to produce the desired outcomes, especially in a complex and dynamic system. Paradoxically, to achieve desired outcomes, managers must pay more attention to the process and less to those outcomes — it is the internal process drivers that we must analyze, with a keen eye for cause and effect.


1. This image was created by David Verble, former Toyota change agent and coach, and Lean Enterprise Institute faculty member, based on the original Purpose, Process, People work by Jim Womack and its later elaboration by Dan Jones.

[For more from the authors on this topic, see "Big Data and Lean Thinking: Balancing Purpose, Process, and People."]

About The Author

Steve Bell's picture

Steve Bell is a Senior Consultant with Cutter Consortium’s Business Technology & Digital Transformation Strategies and Business Agility & Software Engineering Excellence practices. Mr. Bell serves on the faculty of the Lean... Read More