Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Open-source tools keep Twitter up and running

Loek Essers | Nov. 5, 2013
To prevent disruptions and scale up its service while keeping costs down, Twitter has drastically changed its core infrastructure and has adopted some open-source tools.

To curb disruptions and scale up its service while keeping costs down, Twitter has drastically changed its core infrastructure and has adopted some open-source tools.

Twitter processes about 6,000 messages per second -- that's more than 500 million a day and about 3.5 billion a week. The company set a record earlier this year, with 143,000 messages per second during the airing of a movie in Japan, said Chris Aniszczyk, Twitter's head of open-source computing, at a recent conference in Europe.

When Twitter debuted in 2006, it used a monolithic Ruby on Rails application. That worked fine until 2008, when the microblogging service started to suffer a lot of "fail whales" -- the company's term for service disruptions.

The company decided to break the one application it had been using for everything into different services, Aniszczyk said. It also started using open-source tools such as Apache Mesos, a cluster manager; Netty, for creating high-performing protocol servers; and Scalding, for writing big data jobs.

And in another move, Twitter switched its core infrastructure to a Java virtual machine. The end result, said Aniszczyk, is a performance improvement and fewer disruptions.

 

Sign up for CIO Asia eNewsletters.