What’s behind the recent decline and what underlies the discordance?
Companies that strive to become data-driven attempt to replace traditional decision-making forces such as opinion, hierarchy, and intuition with quantitative evidence and statistical analysis. In business contexts this practice is often referred to as Data-Driven Decision-Making, or DDDM. The approach mirrors an earlier shift in medicine towards Evidence-Based Practice (EBP); a movement to replace medical decision-making centred solely on expert opinion. Both EBP and DDDM approaches view quantitative data as foundational. Both benefit from advancements in information systems and technology. Both require new skills for leadership, and substantial change to established cultural norms associated with decision-making. In the medical field, despite inertial forces, the shift towards scientific and statistically-driven care decisions was rapid and widespread (Trinder, 2000). A similarly swift pattern of early adoption was recorded for DDDM.
In 2016, using data collected from the U.S. Census Bureau, Erik Brynjolfsson and Kristina McElheran studied the early diffusion and adoption patterns of DDDM among U.S. manufacturing plants. Their results indicate rapid initial spread. In the five-year span between 2005 and 2010, adoption of DDDM among the organizations they studied nearly tripled; moving from 11% to 30% (Brynjolfsson and McElheran 2016).
Quantitative data regarding this phenomenon is limited for the following five-year period. However, by 2016/2017 survey results suggest a decline and reversal in the adoption DDDM. In a survey of 113 marketers, publishers, advertisers, and developers, the Winterberry Group found that the number of individuals reporting that their data-driven strategies were implemented and delivering results decreased between 2016 and 2017. At the same time, respondents were less likely to identify their organizations as at least “fairly data-centric” than they were they were the year before (The Winterberry Group, 2018). In the years following, the trend continued. In 2017, 2018, and 2019, the percentage of executives who identified their organizations as being data-driven decreased from 37.1%, to 32.4%, to 31% (Davenport and Bean, 2019).
In opposition to this declining trend, public interest and awareness of data-driven decision-making appears to have remained constant over the last five years. Google Trends, which can be used to track the intensity of search query terms over time, indicates consistent or even increasing search intensity for a suite of topics and terms related to data-driven decision-making. (For example, “Data-Driven”, “Data Directed”, “Evidence-Based”, “Business Analytics”, “Data Analytics”, “Business Data”, “Data Insights”)
Over the same time period that search-intensity of DDDM-related terms show neutral or positive trends, business leaders appear less likely to label their organizations as being data-driven. What factors are contributing to the discrepancy? One explanation is that initial enthusiasm contributed to the rapid adoption of DDDM. Then, as the less-than-stellar results of early efforts materialized, enthusiasm dampened. Perhaps in practice, the real-world advantages of DDDM are minimal?
Initial studies, however, contradict this suggestion. Instead, studies provide evidence for tangible benefits of applying DDDM in a corporate context. For example, MIT researchers investigating business practices and IT investment found evidence that organizations employing DDDM achieved a 5-6% productivity and output boost compared to expected performance based on other predictive metrics such as investments and technology usage (Brynjolfsson, Hitt, and Kim, 2011). A similar study investigating the combined and complementary effects of two components of DDDM: big data infrastructure and skilled-labour investment were associated with a productivity boost of 5.9% (Bughin, 2016). A 2014 study of 500 business in the United Kingdom, found that companies that use data more intensively enjoyed an 8% productive advantage compared to those who relied less heavily on data (Bakhshi, Bravo-Biosca, Mateos-Garcia, 2014). The benefits of adopting DDDM appear real and substantial.
If DDDM is delivering, why are executives reporting its use less and less? Perhaps it is not an outright failure of the approach, but a failure to live up to initially inflated expectations. Is it possible that DDDM was over-hyped?
This hypothesis would be consistent with the predictive pattern of the Gartner Hype Cycle. The cycle is commonly used to approximate the maturity of new technologies. However, it can reasonably be extended to offer insight for business concepts such as DDDM, which have strong technological underpinnings. As defined by Gartner, new technological concepts move through phases as they mature. In the first phase a new concept or technology ignites interest and generates media excitement. Enthusiasm builds during this time, but inevitably outpaces early results. In the third phase, as results begin to sink in, discordance develops and enthusiasm wanes. In this period disillusionment and negative publicity deflate expectations. However, through this predictable downturn, early adopters continue to experiment. Finally, through a variety of real-world applications, a more sophisticated understanding of the risks, benefits, costs and advantages associated with a technological concept develops. At this point, adoption begins to increase again; slower this time, but steadily upwards. Is this pattern of maturation consistent with results described earlier for DDDM?
Without true longitudinal data, it’s difficult to say, but the initial, rapid adoption of DDDM described by Brynjolfsson and Kim and the recent decline picked up in expert surveys begin to sketch out a pattern broadly consistent with Hype Cycle dynamics. Supporting this hypothesis, Gartner’s own 2018 Hype Cycle for Digital Marketing and Advertising places “Data Driven Marketing” (an analogous concept) on the downward trajectory, indicating increased disillusionment. Related terms such as “Augmented Analytics”, “Decision Management” are positioned towards a peak of inflated expectations, while other related terms, “Data Preparation” and “Predictive Analytics” are both on the downward slope (McGuire and Yeager; Krensky and Hare, Gartner Research, 2018).
It seems possible that the survey data regarding data-driven success and adoption indicates that disillusionment with the concept is beginning to set in. However, it’s unclear how far along the downward slide the concept is. Has it just begun? Or should we expect a rebound soon? The fact that Google Trend analysis found that search intensity for closely related terms has remained consistent or even increased over the last five years would suggest discordance between the practitioner’s view and the general public’s. This incongruity may indicate that those experienced with DDDM are beginning to register feelings of disillusionment. The sentiment may not yet have spread to the general public. If that’s true, then the peak is just cresting; suggesting a long-slide ahead.
If the hype cycle hypothesis is accurate, perhaps Data-Driven Decision-Making is experiencing growing pains common to technological concepts that move from adolescence to maturity. On the other hand, it’s possible that other external forces are at work. Public anxiety regarding data privacy is increasing (IBM Cybersecurity, 2018) and government legislation is responding. Both the California Consumer Privacy Act (Effective as of January 1, 2020) and the EU’s General Data Protection Regulation (GDPR, active as of 2018) introduce protections regarding the use of personal data. It’s likely that changing legal frameworks, along with more muscular data protection and transparency regimes are causing organization’s to re-think how data is used to make decisions. Perhaps the survey data in question is actually registering an effect related to the increase in the perceived risk associated with data-driven projects? Initially, norms around data privacy allowed near-complete freedom to seek insights from data. Today, organizations rightfully find themselves having to consider privacy, security, transparency, and regulatory compliance before data can be used. Although the earliest survey results discussed here predate high-profile legislative changes, the roots of these discussions stretch back many years. In this view, the down-turn registered in the NewVantage survey (Davenport and Bean, 2019) and others, might indicate a widespread re-evaluation of what it means for an organization to be data-driven.
Whether a natural pattern of maturation, or the result of a shifting regulatory environment, recent survey data suggests a declining trend in DDDM. It is difficult to predict exactly how this trend will play out in coming years. If we extend the hype cycle, then factor in a tightening regulatory environment, it is likely that this decline will continue over the short-term. The current trend is not indicative of a bubble-bursting or a fad collapsing, but rather cooling enthusiasm and collective reflection that may in fact bode well for the future of DDDM.
In many day-to-day cases, neither the reward of getting a decision right, or the cost of getting it wrong, is high enough to justify the extra-time and effort of collecting and analyzing large volumes of data. Practitioners need to be wary of creating culture that requires comprehensive data to make any decision, which could occur if uncritical adoption of DDDM becomes the norm. A better goal would be to foster a culture that reflexively asks, “what’s the most appropriate way to make this decision – expert opinion, personal experience and intuition, or analysis of relevant data?”. Over the long term, adoption of DDDM should increase again, ideally embedded in organizations that use it selectively and thoughtfully.
Credit to Author: Mr. Iain MacKenzie, a former Innovation Consultant with Ayming Canada.
No Comments