The Andon Cord

The origin of the word  “Andon” in Japanese comes from the use of traditional lighting equipment using a fire burning lamp made out of paper and bamboo.  This “Andon” idea was later translated for use in manufacturing in Japan.  The “Andon” became used as a signal to highlight an anomaly (i.e., a flashing light).  This signal would be used to amplify potential defects in quality.  When a defect was suspected, a sign board would light up signaling the specific workstation having a problem.  The signal event would also indicate that the system was stopped due to the defect and was waiting for the problem to be resolved.  The process of stopping a system when a defect was suspected originates back to the original Toyota System Corporation to something called Jidoka.  The idea behind Jidoka is that by stopping the system you get an immediate opportunity for improvement, or find a root cause, as opposed to letting the defect move further down the line and be unresolved.  The Jidoka concept was pioneered by the  original Toyota founder, Sakichi Toyoda.  Sakichi Toyoda is known as the father of the Japanese industrial revolution and also the founder of the original Toyota Systems Corporation (before they manufactured automobiles).  Sakichi Toyoda invented the automatic power loom in 1924 where the loom would automatically stop if a sewing needle broke.  Before his invention, the loom would continue even when the needle was broken.  When this occurred, the downstream process would produce runs in the fabric stitches of the final product. 

Toyota Production Systems

Approximately 25 years later, a gentleman named Taiichi Ohno, who is considered the father of “Toyota Production Systems (TPS)”,  architected a leadership and technology model incorporating some of the original Jidoka ideas along with a lot of other interesting innovations. His ideas produced an unprecedented run of quality and manufacturing success for over 40 years.  To this day, what was done at TPS from around 1950 to 1990 still can’t be repeated in automobile manufacturing, and many have tried.  An abundance of material has been written about TPS, also coined as Lean over the years, and many have tried to copy TPS from that body of knowledge.

Toyota Kata

Mike Rother in his “Toyota Kata” book points out that of the many who try to emulate Toyota, most miss the invisible side of what they were doing.  The key, as Rother suggests, is that it wasn’t the tools that made Toyota great, it was the culture and specific behavior associated behind those tools.  One of the best examples of this was the use of the Andon Cord at Toyota.  The Andon Cord was a manifestation of the original Jidoka principle.  Toyota implemented the Andon Cord as a physical rope that followed the assembly line and could be pulled to stop the manufacturing line at any time. Most western culture analysis of this type of Cord might assume this was implemented as a safety cut off switch.  At Toyota, it was a tool to instill autonomic behavior patterns. This is what Rother calls Kata.

Furthermore, this wasn’t an ask permission to stop the line, the pull actually stopped the line.  As the story goes, anyone could pull the Andon Cord anytime.  Sounds mad doesn’t it?  Salvador Dali eloquently says “The only difference between a madman and me is that I’m not mad.” 

At TPS, the Cord was pulled often.  The mechanics of the Andon Cord were if the Cord was pulled, not only did the line stop, but an “andon” would light up on a signal board to indicate the workstation that was having the issue.  Beyond the mechanics, the culture behind the Andon Cord was a lot more interesting.  The first thing that would happen when the Andon Cord was pulled is that a team leader would immediately “go-see” the issue by visiting the workstation. This was unconditional.  Toyota lived by this “Show Me” culture where in a western culture organization a team lead might put down the coffee on his or her desk and call the workstation to find out what was happening.  This is a key point here.  The “go-see” removes any preconceived notions or potential bias related to the problem.  The “go-see” process is fact based.

High Velocity Edge

In Steven Spear’s “The High Velocity Edge”, he describes a horrifying story of missed opportunities leading up to the 2003 NASA Columbia space shuttle disaster.  The short version of the story is that the thermal protection system on the left wing was damaged just after launch but didn’t become an issue until reentry 19 days later.  After the disaster, an investigation board charged with reviewing the accident found there were at least eight attempted signaling events to notify the crew requesting that they “go-see” the damage.  The first request happened as early as the fourth day of the mission.  These ‘signals’ were not addressed because there was a cognitive bias at NASA regarding this particular issue.  This kind of damage had happened on previous missions and they had always been successful.  Diane Vaughan, a Columbia University sociologist, coined this as “normalization of deviance”.  This is where an organization tends to accept risky anomalies as normal through repeated success.  This form of a blind spot is also referred to as outcome bias.  Outcome bias is where people observe successful outcomes as results as opposed to addressing individual problems at face value.  Sidney Dekker in “The Field Guide to Understanding Human Error” states this kind of the anti Murphy’s Law in that, what can go wrong usually goes right.  Organizations get desensitized due to more positive outcomes than exposed failures. 

Spears suggests some counterfactuals that are intellectually interesting, or at least proposes learning opportunities for the reader.  He suggests the Columbia crew could have been notified and possibly done an extravehicular activity (EVA) inspection to look at the damage. Instead, NASA ignored repeated attempts, andon pulls if you will, to “go-see” the factual conditions.   NASA unfortunately was mired in culture bias’ and preconceived notions of what constituted a real (“go-see”) opportunity.  Maybe if the crew had been notified of just one of the eight known signals, they might have been able to access the damage, and possibly NASA could have done a recovery mission. The meta-point here is that, ironically, NASA didn’t operate like scientists, unlike Toyota’s relentless “go-see” Kata.

Safety Culture

A second important cultural aspect of the “Andon Cord” process at Toyota was that when the team leader arrived at the workstation, he or she thanked the team member who pulled the Cord.  This was another unconditional behavior reinforcement.  The repetition of this simple gesture formed a learning pattern of what we call today “Safety Culture”. The team member did not, or would never be, in a position of feeling fear or retribution for stopping the line.  Quite the contrary, the team member was always rewarded verbally. What Toyota was saying to the team member was “We thank you and your CEO thanks you.  You have saved a customer from receiving a defect.”  Moreover, they were saying, “You have given us (Toyota) an opportunity to learn and for that we really thank you.”   No defect was too small or even if the Cord was mistakenly pulled, the response would never be negative.  In “Toyota Kata”, Rother says that if you look up the word “failure” in the dictionary, it never implies that “failure” is a bad word.  In a classic tayloristic western culture, we tend to shy way from failure at all cost.  We penalize failure and we typically never embrace it.  At TPS, failure was embraced and rewarded.  As Rother also discusses in his book, that at Toyota, at their core, believed failure created learning opportunities.

Improvement Kata

The third important behavior reinforcement of the “Andon Cord” process was that the issue was a priority.  In fact, the second thing that would happen when the team leader would arrive at the workstation after the thank you, is that he or she would ask “How can I help you?”.  An important aspect of this is the “you”.  The incident was not going to be some paper report or bureaucratic long tail process.  The problem was going to be immediately addressed and in fact, the team member who pulled the Cord was the one who was going to fix it.  Another overarching aspect to the culture at Toyota was that everyone had a mentor.  The mentor/mentee relationship was one where it was very important that the mentee understand the problem. Again in western culture organizations, even if a worker felt empowered to stop the line, they might not be inclined to do so if they thought the problem would not get fixed. Even worse, have the issue get bogged down in useless paperwork and never ending meetings.  The team leader at Toyota would then proceed to ask a series of questions trying to drill down to a point where the team member would understand the issue.

Another key point in this process was that even if the team leader knew a better answer, Toyota principally felt that a solution from the team member was a better outcome. In other words, if the problem is not understood at the line level, that is the team member/mentee, then the problem was not solved. This is what I would call learning at scale.

PDCA

Furthermore, the process of solving the issues was controlled by a practice described by Dr. Edwards Deming called Plan Do Change Act (PDCA).  At its core, PDCA is basic scientific method.  Rother refers to this set of tools as Improvement Kata and Coaching Kata.  The Improvement Kata was an iterative mentor/mentee PDCA loop whereas the Coaching Kata was a sort of Socratic dialog between the mentor and mentee.  Again, an important point here is the team member (mentee) would always solve the problem.  It was of high importance that the team member always solve the problem.  Most importantly, it was critical for the team member to understand how the problem was solved.  Otherwise, there would not be in inherent learning and therefore, no real improvement.  Solving problems at Toyota was not the goal, understanding how to solve the problem was. Solving problems in an Improvement Kata mode also creates a second order effect.  By solving one problem, sometimes other second order problems are exposed. 

Alcoa, the aluminum company, set out to have a zero on the job injury policy in 1987.  Paul O’Neal, the new CEO at the time, created a policy that if anyone at Alcoa was hurt on the job, he needed to be notified within 24 hours. This was not slogan based safety.  This was an organizational behavior modification to create a line of sight understanding of why injuries occur at Alcoa.  The results were unprecedented.  However, the interesting side effect of this was that through this rigorous process of forced understanding, they exposed what they called other “pockets of ignorance”.  Their attention to detail for solving their first order problem, safety, surfaced other second order process improvements not necessarily related to safety.

At Toyota, it was important to instill an Improvement Kata based on a PDCA loop.  Plan (P) a countermeasure, implement the countermeasure (D), check or study the results (C), and act on the results either it’s fixed or start the next countermeasure (A).  Imagine all those masked problems that that never get noticed in large complex IT infrastructures due to a possible irregularity of the first order issues and non “go-see” approaches. 

There is a great story in the Toyota Kata book where at one point a particular Toyota plant notices that the average Andon Cord pulls in a shift goes down from 1000 to 700.  As Rother describes most western culture organizations would break out the champagne for such an occasion, not Toyota.  The CEO called an all-hands meeting to address the “problem”.  Notice I said problem.  The CEO then goes on to describe that “we” must have one of two problems here.  One, we are getting lazy and letting more defects get through the system or two, if that is not the case, then we are not operating at our full potential.  He was telling everyone that if they were staffed to handle 1000 pulls per shift, then they should be pulling a 1000 Cords per shift.  The cornerstone behind this kind of thinking is that Toyota had a vision of having 1×1 flow.  This is where there is no inventory build up and work flows freely at every point of the workflow.  Even though 1×1 was sort of a “true north” at Toyota, the CEO was reminding all of them that their Kata should always be pointing towards that direction, what Rother calls the Vision.  In-plane words here, the CEO is saying more pulls equals more learning which means more improvement that gets us towards our vision.  1X1 was a means to an end to say “If we can produce cars faster, cheaper and with higher quality, we win.”

Amazon and the Cord

The Andon Cord has become a metaphor for some modern day Web Scale organizations as well.  Jeff Bezos, the CEO of Amazon, described in a 2013 letter to the Amazon’s shareholders a practice he called the Customer Service Andon Cord.  This was an established practice of metaphorically pulling an Andon Cord when they noticed a customer was overpaying or had overpaid for a service.  Amazon would heuristically scan their systems looking for these kinds of potential customer service mismatches.  These were considered defects at Amazon because they had a vision of being an organization that was always customer centric. They would automatically refund a customer, without the customer even asking, if the service delivery was suboptimal.  I have had this happen to me on a few occasions watching a movie on Amazon Prime, where the next day I received an email telling me they refunded my movie rental cost due to poor quality.  They would also pull the Andon Cord where they found areas where a customer could be saving money.  We see this all the time where Amazon reduces their Cloud Services price even though their service is considered far superior to their closest competitor. This is a form of Kata in practice that drives Amazon towards their stated vision.  In that same shareholders letter, Bezos starts off with a line as follows: “Our energy at Amazon comes from the desire to impress customers rather than the zeal to best competitors”.  It’s no mistake that two of the top 12 books on Jeff Bezos’s recommended reading list are the “Goal” by Eliyahu Goldratt and “Lean Thinking” by James Womack.  In fact, the “Goal” is one of the three books he has all of his top executives read.

The Chaos Cord

Another example of an Andon Cord metaphor used in Web Scale businesses is at Netflix. Netflix has an interesting way of exercising their Andon Cord, although they don’t actually call it an Andon Cord.   Along the same line as Toyota, at Netflix failures are good things.  Netflix has built in their own automated form of Jidoka.  As described earlier, Jidoka is a practice of stopping a process if it breaks.  In the earlier example, the process we described was done via the physical Andon Cord and was a manual process.  At Toyota, there were also automated forms of Jidoka practiced.  Another famous engineer named Shigeo Shingo, who worked with Taiichi Ohno, is credited with the idea of pre-automation.  This is a form of Jidoka that is automatic.  At Netflix, they actually inject this kind of Jidoko into their systems on purpose by intentionally trying to break systems in production.  They have developed what is now famously called Chaos Monkey.  Chaos Money is a process that randomly kills live running production servers.  This behavior is known by everyone who works at Netflix. It’s part of their culture.  There are no surprises about this practice.  Developers plan and Poka-Yoke their code and systems accordingly.  Poka-Yoke is another term that comes from Shigeo Shingo at TPS.  Poka-Yoke means mistake-proofing. 

I was told by Adrian Cockcroft, one of the primary architects behind Netflix’s IT infrastructure, that not knowing about the Chaos Monkey mode coming into a job interview at Netflix was pretty much an immediate no-hire decision.  Imagine that Netflix’s Kata is so obsessed with failure they create their own failures on purpose.  As you can imagine, Netflix is a learning organization and every one of these failures is treated as a science experiment.  They might not literally practice PDCA, but either one of two things happens when a server is killed by their Chaos monkey.  One, they learn that there were dormant defects in the process and fix them, or two, the injected failure was corrected automatically. The best case was where the injected failure caused no customer disruption.  Their Improvement Kata was always moving in that direction.  Like any good operating Kata based organization, Netflix has been practicing their Kata for quite a few years now.  You don’t get to Chaos Monkey overnight.  Much of their Kata has been based on continual learning improvements.   One interesting method or Poka-Yoke,  if you will, is something they do called Circuit Breaker Pattern.  Circuit Breaker Pattern comes from a book called “Release-It” by Mike Nygard.  These are software delivery patterns where the software code is designed very much like a circuit breaker in your home.   If one service dies it isolates or is bounded to only fail the things it controls and not create cascading service outages.  Think about a fuse in your home.  If you inadvertently load up to much power in one area of your house you only wind up losing power in an isolated section. Netflix does a similar implementation for their software services. Their application design is such that one thing breaking should never create cascading failures (like a overloaded circuit/fuse combo in your house).  The main point here is that implementing an Andon Cord in an organization is not something you do overnight.  It takes a continuous improvement roadmap to get there and must have behavior reinforcement built into the process.  It takes a fierce commitment and practice of improvement (Improvement Kata) and an equally skilled leadership coaching approach (Coaching Kata).  If you want to investigate the concept of Kata more deeply, I highly recommend reading Mike Rother’s “Toyota Kata”.

- About The Authors

John Willis

John Willis has worked in the IT management industry for more than 35 years. Currently he is an Evangelist at Docker Inc. Prior to Docker Willis was the VP of Solutions for Socketplane (sold to Docker) and Enstratius (sold to Dell). Prior to to Socketplane and Enstratius Willis was the VP of Training & Services at Opscode where he formalized the training, evangelism, and professional services functions at the firm. Willis also founded Gulf Breeze Software, an award winning IBM business partner, which specializes in deploying Tivoli technology for the enterprise. Willis has authored six IBM Redbooks for IBM on enterprise systems management and was the founder and chief architect at Chain Bridge Systems.

Follow John on Social Media

• 
•