There will be instances that demand the fast response times of dozens of machines in a Hadoop cloud running in parallel, but many will do fine plugging along on a single machine without the hassles of coordination or communication.
It’s not so much that Hadoop is cooling off. It’s more that Spark is red hot, making the Hadoop model look a bit old. Spark borrows some of the best ideas of Hadoop’s approach to extracting meaning from large volumes of data and updates them with a few solid improvements that make the code run much, much faster. The biggest may be the way that Spark keeps data in fast memory instead of requiring everything be written to the distributed file system.
Of course many people are merging the two by using Spark’s processing speed on data stored in Hadoop’s distributed file system. They’re more partners than competitors.
Hot: Artificial intelligence/machine learning
Not: Big data
No one knows what the phrase “artificial intelligence” means, and that helps the marketers, especially since the term “big data” has run its course. They’re grabbing terms from artificial intelligence and upgrading the sophistication of the big, number-crunching algorithms that plow through our log files and clickstreams. By borrowing the more sophisticated algorithms from the 50-odd years of AI research, we stand a better chance than ever of finding that signal in the noise. Tools run the gamut from machine learning frameworks to cognitive computing, all the way up to IBM’s Watson, which you can now ping to solve your problems. Each offers its own level of machine intelligence, and with this, the promise of taking over more of the data analysis and forensics for us.
Hot: Game frameworks
Not: Native game development
Once upon a time, game development meant hiring plenty of developers who wrote everything in C from scratch. Sure, it cost a bazillion dollars, but it looked great. Now, no one can afford the luxury of custom code. Most games developers gave up their pride years ago and use libraries like Unity, Corona, or LibGDX to build their systems. They don't write C code as much as instructions for the libraries. Is it a shame that our games aren't handcrafted with pride but stamped out using the same engine? Nope. Most of the developers are relieved -- because they don't have to deal with the details, they can concentrate on the game play, narrative arc, characters, and art.
Hot: Single-page Web apps
Remember when URLs pointed to Web pages filled with static text and images? How simple and quaint to put all information in a network of separate Web pages called a website. The design team would spend hours haggling over the site map and trying to make it easy enough to navigate.
Sign up for CIO Asia eNewsletters.