The desire to make better decisions faster is one of the fundamental drivers of new big data analytics technologies and a general push toward data-driven decision-making. But the relationship between data and intuition — the old 'gut feeling' — is a complicated one, says Peter Swabey, senior editor, technology at the Economist Intelligence Unit (EIU), the research and analysis division of The Economist Group.
"They both play a role," Swabey says. "The process of developing data is the process of trying to identify what the true state is. In identifying that, your intuition could be a useful guide."
In an effort to better understand how business decisions are made, predictive analytics firm Applied Predictive Technologies (APT) asked the EIU to conduct a study, resulting in a report released this week: Decisive action: How businesses make decisions and how they could do it better.
In February 2014, EIU surveyed 174 senior managers and executives from around the world and from a range of industries, 49 percent of whom represented organizations with more than $500 million in annual revenue. EIU also conducted in-depth interviews with practitioners and experts.
The study found that 42 percent of respondents said they collect and analyze data as much as possible before making a decision, while an additional 17 percent said they approach decisions empirically by developing hypotheses and performing tests to prove or disprove them. In other words, 59 percent rely on data to help them make decisions. Of the remainder of respondents, 32 percent said they seek to collaborate on decisions as much as possible, while only 10 percent said they primarily rely on intuition to make decisions.
"Despite the apparent popularity of data-driven decision-making, however, intuition is in fact valued highly," says Jane Bird, author of the EIU report. "Nearly three-quarters of respondents (73 percent) say they trust their own intuition when it comes to decision-making. Even among the data-driven decision-makers, over two-thirds (68 percent) agree with that statement."
And, Bird notes, 68 percent of them would be trusted by their peers and superiors to make a decision not supported by data.
"However much data you take in, and whatever the interview process, when you get to the end there has to be an element of gut feel too," says Alison Robb, group director for people, customer, communications and commercial at Nationwide Building Society, one of the expert practitioners interviewed for the EIU report. "It's partly chemistry, experience and knowing what you do and don't like."
Intuition Can Help Identify Problems With Data Analysis
Perhaps nothing illustrates the inextricable link between data-driven decisions and human intuition better than how people would respond if the available data contradicted gut feeling. Fully 57 percent of survey respondents said that if confronted with that situation, the first thing they would do is reanalyze the data; 30 percent said they would collect more data and only 10 percent said they would take the course of action suggested by the data.
Sign up for CIO Asia eNewsletters.