Microsoft added three new big data services to its Azure cloud platform yesterday as part of its ongoing efforts to make Azure a leading platform for big data services and to make it ready to tackle the Internet of Things (IoT).
"Every day, IoT is fueling vast amounts of data from millions of endpoints streaming at high velocity in the cloud," says Joseph Sirosh, corporate vice president of Machine Learning at Microsoft. "Examples of streaming analytics can be found across many businesses, such as stock trading, fraud detection, identity protection services, sensors, web clickstream analytics and alerts from CRM applications. In this new and fast-moving world of cloud and devices, businesses can no longer wait months or weeks for insights generated from data."
Two of the services -- Stream Analytics and Azure Event Hubs -- are intended to help customers process data from devices and sensors in the IoT. Meanwhile the Azure Data Factory service is about information production by orchestrating and managing diverse data, especially in hybrid environments where some data sources reside in the cloud and others are on-premise.
Stream Analytics is an event-processing engine, akin to the Apache Storm framework, except Stream Analytics is a managed service. Microsoft announced support for Apache Storm in Azure HDInsight earlier this month.
"The scenarios are similar," says Herain Oberoi, director of product management, Microsoft Data Platform, at Microsoft. "Stream Analytics is more of a managed service whereas Storm is an additional project in the Hadoop ecosystem that you have to manage yourself. For customers that want a more managed service where the actual coding is less, the package management is less, then Stream Analytics makes more sense. We're investing in both."
Stream Analytics is meant to work hand-in-hand with Azure Event Hubs, a scalable service for collecting data from millions of events per second.
"With Azure Stream Analytics, businesses can gain insights in real-time from data generated by devices, sensors, infrastructure, applications and other sources," Sirosh says. "Developers can combine streams of data -- such as clickstreams, logs, metering data or device-generated events -- with historic records or reference data. Complementing Stream Analytics, Azure Event Hubs is a highly scalable publish-subscribe ingestor that collects millions of events per second, allowing users to process and analyze data produced by connected assets such as devices and sensors. Stream Analytics provides out-of-the-box integration with Event Hubs -- when connected, these two solutions enable customers to harness IoT by processing and analyzing massive amounts of data in real time."
Beyond the Internet of Things
However, Microsoft isn't focusing on only the Internet of Things. The Azure Data Factory service is intended to address a more fundamental challenge: effectively managing, coordinating and processing data from many different sources -- across geographic locations, on-premises and cloud, whether the data is structured or unstructured -- while dealing with shifting business requirements.
Sign up for CIO Asia eNewsletters.