Can Your Culture Handle Experimentation?

March 10, 2020

Can Your Culture Handle Experimentation?

To successfully innovate, companies need to build a culture that invites experiments at a large scale, even when budgets are tight, and it’s never been cheaper and easier to do so. Computer simulations, A/B test tools, and other approaches enable teams to ask an endless stream of “what if” questions. Indeed, as companies build experimentation capacity they often find that the bottleneck to running more experiments shifts from technological difficulties to ones of culture. To build a true experimentation culture—including the shared behaviors, beliefs, and values for large-scale and broad-scope testing—leaders need to ensure that their culture has seven attributes: 

  1. a learning mindset, 
  2. consistent rewards, 
  3. intellectual humility, 
  4. strong ethics, 
  5. a trust for tools, 
  6. an appreciation for exploration, and 
  7. an ability to embrace new models of leadership. 

Attribute 1: A Learning Mindset 
Experimenting with many diverse ideas is crucial to innovation. But when people experiment more rapidly and more frequently, inevitably most tests fail. Early failures are not only desirable, but also necessary, since experimenters can quickly eliminate unfavorable options and refocus their efforts on more promising alternatives. In other words, an experiment that fails is not a failed experiment or a waste of time. If anything, a low failure rate indicates that employees are reluctant to take risks. 

That same learning mindset can be applied even when you are not consciously trying to experiment. A happy accident can lead to a surprising insight, which the team could optimize. Savoring surprises and actively seeking them out, both as a starting point and outcome of an experiment, is at the heart of an experimentation culture. 

Managers who overemphasize winning due to a concern about wasting time on low-yield experiments may inadvertently encourage employees to focus on familiar solutions and short-term gains. Studies have shown that short-term losses in experimentation can facilitate innovation and performance in the long run. Employees who prefer to work on activities in which failure is more likely also tend to persevere when times get tough, engage in more challenging work, and perform better than their safety-seeking peers. 

Failures should not be confused with mistakes. Unlike failures, mistakes produce little new or useful information and are therefore without value. Distinguishing between failures and mistakes is desirable, but even the best organizations often lack the management systems necessary for carefully making that distinction: to simultaneously promote failures and weed out mistakes. 

Attribute 2: Rewards Are Consistent with Values and Objectives 
Creating a culture that takes full advantage of new experimentation tools requires a deeper understanding of the factors that affect experimentation behavior. 

Giving mixed signals, such as encouraging employees to experiment while maintaining a reward system that punished failure made their performance worse, leading to less experimentation—even less than when we consistently discouraged them from experimenting. This demonstrates the danger of overly simplistic management interventions, where changing only one factor (values or rewards) can lower employee performance. 

When teams take on the dual objective of meeting day-to-day business demands and running experiments, incentives can get misaligned. 

Attribute 3: Humility Trumps Hubris 
Experimental results can go against entrenched interests, beliefs, and cultural norms. When they do, people often respond with a reflexive rejection of the results. We tend to happily accept “good” results (“wins”) that confirm our biases because it feels good. But we challenge and thoroughly investigate “bad” results (“losses”) that go against our assumptions. And when organizations run large-scale experimentation programs, they must disconfirm beliefs at a very rapid rate. This puts stress on managers and can move organizations past their breaking point. 

Experiments that lead to new insights but lack an accepted theory of cause-and-effect run a high risk of encountering bias and rejection. Conversely, a fundamental understanding of cause, backed with testable predictions, can lead to acceptance and change. 

Human bias may not be as readily observable as an outright rejection. At times, bias can be very subtle and creep into the experimental process. To offset bias, conscious or unconscious, in large-scale experimentation programs, companies should promote full transparency: all employees get access to all experimental protocols and data. Sunlight is the best disinfectant. 

For an organization to fully absorb experiments, unbridled curiosity needs to drive out strong opinions and biases. That’s especially true for higher status individuals—the bosses—whose promotions most certainly involved some lucky business decisions. When it comes to novelty, even the bosses can be wrong. 

Attribute 4: Experiments Have Integrity 
Clearly, the debate of what’s ethically meaningful is important for businesses. But the ethics question of what should and should not be reviewed has to be carefully weighed against the opportunity cost. Too much internal scrutiny may slow experimentation to a trickle. Too little scrutiny may lead to another “emotional contagion”–like blow-up. 

Before conducting a test, stakeholders must agree that the experiment is worth doing. That needs to include the perception of its integrity— the goodness or badness of an experiment. 

Attribute 5: The Tools Are Trusted 
In the excitement of imagining how much improvement is possible, companies can easily forget that these tools don’t create products and services or lead to better decision-making all by themselves. In fact, when incorrectly integrated into an organization (or not integrated at all), new tools can actually inhibit performance, increase costs, and cause innovation to founder. In a nutshell, tools are only effective if people and organizations trust and use them.

As painful as it is for companies to invest in new experimentation tools, doing so, in fact, is the easy part. Far more difficult is how to trust and use them effectively—to manage “tools in use.” This requires paying attention to human involvement in running experiments and, at times, resisting the temptation of automating every single step. 

Attribute 6: Exploration and Exploitation Are Balanced 
Building an experimentation culture that can balance the paradox of failure and success, experimentation and standardization, and ultimately, long- and short-term business pressures is neither easy nor straightforward. This tension between creating value through innovation—to explore— and capturing value through operations—to exploit—is at the heart of running a successful business. Finding the right balance will challenge senior management. 

Today, firms that have focused on exploitation have found that the very factors that make them successful are also inhibiting their ability to explore. Process standardization and efficiency can get in the way of learning from failure, experimentation and innovation. This calls for interventions from a company’s leadership. 

Attribute 7: Ability to Embrace a New Leadership Model 
When it comes to innovation, being too frugal can have its drawbacks, particularly if the result is that a company’s pipeline of new products and services begins to dry up. And that’s the danger facing extremely efficient businesses that value standardization, optimization, and low variability. They leave themselves vulnerable to underinvesting in experimentation. 

All this, however, raises a tricky issue about top management: If all major decisions are to be made through experimentation, then what’s the job of senior leaders? 

True experimentation organizations embrace a new leadership model. Instead of viewing leaders primarily as decision makers, the model encompasses three important responsibilities. First, a senior executive’s job is to set a grand challenge that can be broken into testable hypotheses and key experiments. Second, they need to put in place systems, resources, organizational designs and standards (e.g., tools, program management, skills training) that allow for large-scale, trustworthy experimentation. And third, executives need to be a role model for all employees. That means living by the same rules as everyone else.

Reprinted by permission of Harvard Business Review Press. Adapted from EXPERIMENTATION WORKS: The Surprising Power of Business Experiments by Stefan H. Thomke. Copyright 2020 Stefan H. Thomke. All rights reserved.

The Authors: 

Stefan Thomke is the William Barclay Harding Professor of Business Administration at Harvard Business School and author of the books Experimentation Matters: Unlocking the Potential of New Technologies for Innovation (Harvard Business Review Press, 2003) and Managing Product and Service Development (McGraw-Hill/Irwin, 2006).