3 Tips for Generating a Performance Bonus Through Diversity
Lessons from the Netflix Prize.
Posted Dec 04, 2019
Hiring diverse team members can create a performance bonus when the team is addressing a complex task. Such “diversity bonuses” in teams are generated by members’ non-overlapping cognitive diversity, which includes diverse experience, knowledge, mental models, representations, tools and so on.
To illustrate diversity bonus, consider how Netflix made an unprecedented improvement in its algorithm to predict users’ ratings by organizing a well-designed open competition. In 2006, Netflix released a dataset of 100 million anonymous movie ratings, and announced an open competition to develop an algorithm that would increase the accuracy of the company’s recommendation system.
To win the prize of $1 million, participants had to beat Netflix’s current prediction algorithm, Cinematch, by 10 percent. Team BellKor won the 2007 Progress Prize by developing 50 models, and basing its winning prediction on an ensemble of these models. However, this was insufficient to win the ultimate prize, as the team was only able to improve on Cinematch by 8.4 percent. Therefore, it sought help.
In 2008, BellKor merged with another team, Big Chaos, whose strength lay in determining an optimal weighting when combining the predictions of different models. The merged team won the 2008 Progress Prize, but the improvement still failed to reach the 10 percent threshold.
In 2009, a new partner, Pragmatic Theory joined the ensemble. Like Big Chaos, Pragmatic Theory did not produce a model that could outcompete those developed by BellKor alone, but it did provide a novel behavioral insight that no contestant had previously considered: The same individual may rate the same movie differently depending on the context. For example, ratings may differ if made immediately after watching a movie or later on, or after watching it on a weekday rather than at the weekend. The merged team, BellKor’s Pragmatic Chaos finally beat the threshold and won the Netflix Prize with a prediction that improved on Cinematch by 10.06 percent.
This case illustrates how enhancing teams’ cognitive diversity may generate a performance bonus under the right circumstances, because heterogenous teams may outperform homogeneous teams when addressing complex tasks. However, this “diversity bonus” is often unfulfilled owing to various barriers that deter organizations from appreciating and engaging with candidates or ideas that deviate from the status quo.
Netflix’s open competition introduced at least three features that enabled the firm to overcome these barriers:
First, the competition was based on a sufficiently difficult target, which is necessary for a diversity bonus to occur. Diversity bonuses are relevant only to sufficiently complex tasks, which must involve more dimensions or variables than any individual or naturally assembled team can master. For example, if the Netflix Prize had been based on an easier target—say, a five percent threshold—the most able team, BellKor, would have won without help from the other two teams, leaving the diversity bonus unfulfilled. The diversity bonus is most relevant when a task is sufficiently complex: No individual or self-organized team (subject to network-induced similarities) should be expected to have sufficient cognitive capacity to master all the knowledge, perspectives, and tools essential to address the task.
The second important feature is that Netflix opened up by engaging various specialist groups, rather than recruiting diverse teams in-house. The winning team (and its winning output) would have been unlikely to be assembled or realized if Netflix had not organized this crowdsourcing competition. BellKor was formed by three seasoned data scientists at the AT&T lab. The 50 models they developed reflected a high level of expertise in recommendation systems, as well as rich cognitive diversity within the team, with each model representing an idiosyncratic set of assumptions about raters’ behaviors. Combining these diverse models created substantial diversity bonuses in predictions, but not yet enough to beat the difficult threshold.
Then came team Big Chaos, which was less able than BellKor as measured by its lower predictive accuracy, but still provided a useful addition to Bellkor’s cognitive repertoire. The members of Big Chaos were computer scientists whose work focused on ways to combine model predictions with sophisticated algorithms. By merging with Big Chaos and sharing all the model details and insights, the updated predictions fully realized the potential of BellKor’s diverse models.
However, the eventual outcome was determined by a contribution from an unexpected source. Pragmatic Theory was formed by two amateurs who had experience in neither data analytics nor computer science, but provided an insight overlooked by most experts. Only then did the merged team manage to beat the 10 percent threshold by integrating state-of-the-art knowledge about recommendation systems (from BellKor), a novel combinatory tool (from Big Chaos), and an insight from “outside the box” (from Pragmatic Theory).
Finally, a diverse team may be assembled but its diversity may fail to be fully mobilized. The third feature of the Netflix Prize is that it introduced a suitable incentive scheme to encourage teams to compete as well as to collaborate. Recognizing that a diversity bonus requires a strong incentive to mobilize teams’ stock of diversity to improve performance, the Netflix Prize presented a winner-takes-all set-up. The most able team, BellKor was motivated to work with teams very different from itself; without them, its ability alone would have won it nothing. Interestingly, after Bellkor’s Pragmatic Chaos announced that it had improved on Cinematch by 10.06 percent, 30 other teams were combined to form Team Ensemble, which produced an identical 10.06 percent improvement. However, BellKor’s Pragmatic Chaos won because it submitted its codes 22 minutes before Ensemble in the final round.
In sum, enhancing cognitive diversity in teams may generate a performance bonus under the right conditions because heterogeneous teams may outperform homogeneous teams when addressing complex tasks. The Netflix Prize case highlights three important tips for generating diversity bonus: (1) the need for the task to be sufficiently complex; (2) the need for the team to be sufficiently diverse in terms of its collective cognitive repertoire (not necessarily identity diversity such as gender or race etc); (3) appropriate incentives and communication mechanisms must be in place to encourage diverse ideas to be shared, integrated, and evaluated.
This case illustrates how a well-designed structure can enable diversity bonus to be captured. It also suggests why potential diversity bonuses remain unfulfilled in other contexts, awaiting those who appreciate the logic of generating diversity bonus to exploit untapped opportunities.