Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Which freaking big data programming language should I use?

Ian Pointer | April 1, 2016
When it comes to wrangling data at scale, R, Python, Scala, and Java have you covered -- mostly

You have a big data project. You understand the problem domain, you know what infrastructure to use, and maybe you've even decided on the framework you will use to process all that data, but one decision looms large: What language should I choose? (Or perhaps more pointed: What language should I force all my developers and data scientists to suffer?) It's a question that can be put off for only so long.

Sure, there's nothing stopping you from doing big data work with, say, XSLT transformations (a good April Fools' suggestion for tomorrow, simply to see the looks on everybody's faces). But in general, there are three languages of choice for big data these days -- R, Python, and Scala -- plus the perennial stalwart enterprise tortoise of Java. What language should you choose and why ... or when?

Here's a rundown of each to help guide your decision.


R is often called "a language for statisticians built by statisticians." If you need an esoteric statistical model for your calculations, you'll likely find it on CRAN -- it's not called the Comprehensive R Archive Network for nothing, you know. For analysis and plotting, you can't beat ggplot2. And if you need to harness more power than your machine can offer, you can use the SparkR bindings to run Spark on R.

However, if you are not a data scientist and haven't used Matlab, SAS, or OCTAVE before, it can take a bit of adjustment to be productive in R. While it's great for data analysis, it's less good at more general purposes. You'd construct a model in R, but you would consider translating the model into Scala or Python for production, and you'd be unlikely to write a clustering control system using the language (good luck debugging it if you do).


If your data scientists don't do R, they'll likely know Python inside and out. Python has been very popular in academia for more than a decade, especially in areas like Natural Language Processing (NLP). As a result, if you have a project that requires NLP work, you'll face an embarrassing number of choices, including the classicNTLK, topic modeling with GenSim, or the blazing-fast and accurate spaCy. Similarly, Python punches well above its weight when it comes to neural networking, with Theano and Tensorflow; then there's scikit-learn for machine learning, as well as NumPy and Pandas for data analysis.

There's Juypter/iPython too -- the Web-based notebook server that allows you to mix code, plots, and, well, almost anything, in a shareable logbook format. This had been one of Python's killer features, although these days, the concept has proved so useful that it has spread across almost all languages that have a concept of Read-Evaluate-Print-Loop (REPL), including both Scala and R.


1  2  3  Next Page 

Sign up for CIO Asia eNewsletters.