Building Fault Tolerance

Last week my ten-year-old son got a question wrong on his math test. No big deal, right? But I saw his face fall as he realized it was the end of his perfect streak. However, I’m actually glad for the small slipup because I know it’s important that he become adept at learning from mistakes.

Like my son, I often obsessed about my faults. I could do a thousand things right, but my one blunder would grate on me. That singular bad grade. The flubbed line in a speech. An error in analysis. These would color my entire memory of an event.

To this day, I recall how I mispronounced “TNT” when reading aloud in second grade. I remember doing poorly on a high school history exam because I didn’t know the definition of “oligarchy.” And I recollect the errors I made when collecting data during a summer internship, weeks wasted in the lab because I didn’t have the frequency measurement set up correctly.

But just as I recall my disappointment and shame at these gaffes, I also remember the lessons I learned. And I assure you I will never make those same mistakes again.

Over the years, I’ve become more resilient, better at picking myself up, dusting myself off, and trying again. Asking: What went wrong? What could I do differently? What should I try next? I’ve become more fault tolerant.

A fault-tolerant system performs without interruption even if one or more of its components fail. It is designed to compensate for failures by automatically detecting issues and immediately correcting or failing over to a backup component. The mainframe is a great example of this; if a processor core or memory node fails, the system will switch over to a redundant core or node and continue to operate as if nothing went wrong.

So, what does it take for individuals and institutions to become fault tolerant?

Embrace Experimentation. The concept of “experimentation” is one that recognizes no one has all the answers but that it is possible to discover them. Individuals who embrace experimentation are genuinely curious: they question, they learn, they imagine, they get things wrong, and then they figure things out. This spirit is embodied in the interesting phrase “everything is figureoutable,” which is also the title of a recent book by Marie Forleo. An organization with a culture of experimentation applies a rigorous methodology to finding answers. The scientific method provides a structured approach to discovery that has stood the test of time: question, study, hypothesize, test, assess, report. Based on results you might modify your hypothesis and repeat the whole process again and again. These are the tools of innovation, which fueled discoveries like renewable energy, the internet, and COVID-19 vaccines. Experimentation is a powerful tool in business too. Take 3M, for instance, which ran dozens of laboratory and in-market experiments before eventually launching its successful Post-It Notes product line. Many companies today embrace “A/B testing” of marketing and web content, experimenting to find which messages, layout, or graphics appeal to consumers and drive highest click-through rates and sales.

Avoid the Nirvana Fallacy. Don’t try to be perfect. Just don’t. It’s an impossible ideal that no individual, team, or organization can possibly live up to. “Perfect is the enemy of good” is an aphorism often attributed to the French writer Voltaire. This recognizes that often a good enough solution is sufficient — and arrived at much faster — than a perfect solution, and that striving for perfection can stand in the way of progress. For the straight-A student or the undefeated team, this is a hard lesson to grasp. Of course, a focus on achievement, quality, and winning is admirable. But does it really matter how the dishwasher is loaded? Will that wrong answer on a fifth-grade math test change the course of my son’s life? An undefeated team doesn’t know how to respond when faced with an impending loss; they haven’t exercised the “comeback” muscle, as Gonzaga demonstrated last week. So too an individual who has always been praised, always achieved their goals, may struggle to adapt when faced with adversity. In technology, we adopt the concept of a minimum viable product (MVP) with “just enough” features and capabilities to validate the concept, to get client and market feedback. It’s not perfect — by its very definition — but it’s a start, and we iterate, based on feedback, to make it better and better.

Adopt a Growth Mindset. The real world is messy, complex, and ever-changing. Today’s technology wasn’t even invented when we were kids; history is being re-written every day. So those who are stuck in an “everything-I-need-to-know-I-learned-in-school” mentality will be left behind. They won’t be adaptable, able to respond to the next challenge, excited to tackle the unknowns. People who embrace a growth mindset — first described by Stanford psychologist Carol Dweck — recognize that what you know is less important than what you can learn. They demonstrate a desire to understand, prioritize continuous improvement, and see failure as a springboard for growth. Dweck describes how a growth mindset creates a passion for learning rather than a hunger for approval.

Fail Fast. Becoming tolerant of failure ultimately means failing more often and more quickly — in small but important ways. IBM’s Thomas Watson, Sr. famously said, “The fastest way to succeed is to double your failure rate.” He recognized the importance of encouraging risk taking and that businesses must have a certain tolerance for failure. That means leaders cannot penalize individuals or teams that try new things; there is no surer way to stifle innovation than this. Yet we don’t want to make big, costly, reputation-damaging, business-critical mistakes. The answer: small, iterative mistakes — lots of them — that allow course corrections and learning. This is, of course, the basis for agile iterative development: bite-sized chunks of work, with interim checkpoints, allowing for many tiny missteps but ensuring no big blunders.

Learn From Mistakes. What happens when you make a mistake? Do you shut down, give up, or sulk in the corner with your figurative tail between your legs? Do you feel embarrassed, guilt-ridden, or vulnerable? Or do you ultimately recognize failure for what it is: an opportunity to learn and deepen knowledge? It can be difficult to truly embrace this, particularly because behaviors taught and reinforced in childhood often run counter. Admitting mistakes takes courage, taking responsibility shows humility, and embracing learning shows wisdom. Mistakes are life lessons; they teach us to improve and ultimately make us better. Learning from mistakes is essential for a team too. Take the time to reflect: what went wrong, what could have gone better, what will we do differently going forward?

Here’s the thing: We all make mistakes. Nobody is flawless. Not CEOs, not Nobel Prize winners, not presidents, supermodels, activists, celebrities, or philanthropists. No one. Everyone is on a continuous improvement plan, and we must all get used to being perfectly imperfect.