Although there's a strong trend toward analysing data in memory and delivering insights in real time - to inform "sense and respond" systems - don't imagine that the world is going all real-time. Instead, Jeff advises that you should view batch approaches to analysis as an important complement, as delivering "periods of reflection" that can deliver insights that you can then use to improve the accuracy and usefulness of the model that drives your "sense and respond" systems. Jeff labels these two sides of the analytical world with catch phrases: "sense and respond" (relevance finds the sensor) vs. "explore and reflect" (relevance finds you). Jeff advises we use both sides together, which should inform future architectures for doing advanced analytics.
In contrast, today we do analytics in stovepipes - we have one set of algorithms to analyse structured data, different algorithms for unstructured data, and still more (different) algorithms for social data! Jeff believes that in the future we must take a more integrated approach to analytics, with algorithms that reason over datasets that mix all types of data, and link them all. It's only through this broader view that we can do what casinos do, and catch the bad guys while they are still playing Blackjack.
What This All Means For You
Below find my take on how you should act upon Jeff Jonas' insights, but I also urge you to engage with Forrester's analysts who spend every waking moment thinking about business intelligence, Big Data, and the potential for deeper business insights that these and other innovations can bring:Boris Evelson,Martha Bennett, Mike Gualtieri, Noel Yuhanna, Michele Goetz, Brian Hopkins, and others. In my view::
- Integrate your analytics stovepipes. Gaining deep insight requires a more integrated approach to analytics, bringing together all sources of information, whether structured, semi-structured, or unstructured (including media) into one pool of observations for analysis. This runs counter to the current practice in many organisations of more stove-piped approaches to analytics, so will require a major upheaval to accomplish, but it will be worth it for those that most require this kind of intelligence. The implications impact organisation, staffing/skills, choices of technology, and architecture.
- Integrate real-time and batch analytics for deeper insight. Both real-time and batch approaches are critical, and are also more complementary than many people realise. Although the need to act quickly on information that develops in real time (sense and respond) is the primary driver of the need to increase investments in real-time, the opportunity to inform batch analyses/models with new insights that are constantly emerging from real-time channels is an under-recognised source of added value that can help support the business case for real-time, just as insights from "deep reflection" via batch methods can inform and improve "sense and respond."
- Don't be afraid of real-time. I was struck by Jeff's view that real-time may not cost more, as many expect it does. My own research, talking to people who are doing new work in-memory and using new technology like SQLStream or Streambase, or CEP, suggests that Jeff is right, that these innovative new ways of gaining insight often develop those insights much more efficiently than through other approaches that require cranking through the whole haystack, instead of reaching in and picking out just the needle you care about.
- You need the right people to gain these insights. Transforming your approach to analytics will depend mainly on having the right people - as Jeff put it, you should hire "curious" people. In the future it will be more important for an analyst to be curious, even driven, than for the analyst to know SQL. These curious people will be seeing the emerging picture uncovered as data finds data - algorithms discover connections among many different observations - and using those insights to continually refine their analytical models and augment their sources with additional observations.
- Beware the privacy and regulatory implications of integrating analytics. The value of combining information from multiple sources will motivate organisations that urgently require better insights from this data to consider how to obtain insights from the datasets they need without violating policies and regulations designed to protect the interests of citizens, while staying away from the legal jeopardy of a "fishing expedition."
Sign up for CIO Asia eNewsletters.