Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Big data log analysis thrives on machine learning

James Kobielus | June 23, 2014
Huge quantities of log data generated by all sorts of devices opens immense potential for insight, but machine learning is needed to make sense of it

Another key use of unsupervised and reinforcement learning is to identify significant patterns that either never occurred before or, if they had, never been flagged by human analysts as anything other than "noise." The article's authors discuss a hypothetical security-log analysis application of machine learning that can "immediately spot an atypical access pattern for a user, even if that specific access pattern had never been seen before, and prevent particularly high-risk losses of private information."

Many of the most disruptive insights from massive log data will be of this nature: complex, buried, and unprecedented. Learning from the log data itself, rather than from any a priori knowledge, will be how many data scientists spend much of their time. They will increasingly tune their machine-learning algorithms to listen for "signals" in the log that even the most advanced human subject-matter experts had previously overlooked.

Source: InfoWorld

 

Previous Page  1  2 

Sign up for CIO Asia eNewsletters.