CARFAX, the online vehicle tracking and valuation website, built its first database in 1984 based on technology named OpenVMS. At the time, it was cutting edge for its ability to handle millions of records.
But the company grew. "We had a hard time scaling it and finding people to work on OpenVMS," says Jai Hirsch, senior systems architect for data technologies at CARFAX. The company needed a new database.
For years the default enterprise databases have been based on SQL, a programming language that databases from Oracle, Microsoft, SAP and many other companies predominantly use. But increasingly, SQL databases aren't an ideal fit for companies like CARFAX. Traditionally, SQL databases are based on rows and columns; CARFAX has 13.6 billion records associated with 700 million vehicles. A column-based system would have required thousands of columns and tabs, but for any given vehicle maybe only a dozen of them would be populated. It just wasn't ideal for CARFAX.
Hirsch and his team began playing with a new breed of databases referred to as NoSQL, which originally meant "Not only" SQL databases. NoSQL is a small but growing segment of the database market, according to 451 Research's Matt Aslett, who predicts it at about 2% of the size of the SQL market. While nascent, it's an important segment, especially for companies with big data needs that don't fit well into traditional SQL databases.
CARFAX is now running a 108-server deployment of MongoDB, one of the leading document NoSQL databases. It holds 10.6TBs of data and 1.5 billion vehicle documents are added a year. The Mongo database processes five times as many records per second as CARFAX's legacy system. Its working great for Hirsch and CARFAX and Aslett says it could be coming to many more enterprises in the near future.
Experts say that despite the rise of NoSQL databases during the past three to five years, NoSQL is not necessarily a replacement for SQL databases. Aslett says NoSQL databases are generally used for new projects and for handling data sets that can strain SQL systems. NoSQL databases have less rigid schemes, allowing more flexibility in how databases are created, used and managed. But SQL databases aren't going away. They're stable, proven and well-known amongst database administrators and they're still very good at transactional data that many legacy systems heavily rely on.
Forrester big data analyst Noel Yuhanna says NoSQL databases are "complementary" tools to SQL that "fills gaps found in traditional database systems."
Much of the movement to NoSQL is being driven by megatrends happening in the enterprise IT market. For example, developers are pushed to produce more applications faster. Having a database that's more flexible, can be configured multiple different ways and changed on the fly (NoSQL systems shine at this) can be an advantage in these faster-paced development environments. Meanwhile, applications being created are often interactive, maybe dealing with social streams and other data that doesn't fit neatly into a SQL database.
Sign up for CIO Asia eNewsletters.