Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Why Dropbox left Amazon's cloud and built its own from scratch

George Nott | June 28, 2016
Cloud storage service migrated 500 petabytes without users noticing as part of its ‘Magic Pocket’ project

One in two Australian Internet users in Australia use Dropbox. Worldwide, more than 1.2 billion files are saved to the cloud storage service every 24 hours. And until recently, all those files were sitting in Amazon Web Services' public cloud.

In 2013 Dropbox decided it would store them itself. Over the last three years the company has been busy building its own storage infrastructure and migrating 500 petabytes of data from Amazon's cloud.

"This has been, as far as I know, the largest ever migration of a company from cloud services into managed infrastructure," says Dropbox principal engineer and storage team lead James Cowling. "And happening at a time when the trend is definitely in the other direction. It's a big investment. It's a pretty bold investment too."

The company had become big enough for the move to make economic sense. And with an eye on the enterprise market, it wanted to release slick and speedy productivity and collaboration tools. For that it needed to boost performance of its storage infrastructure.

"Eventually it became very clear that storage is core to our business, and we can provide a better experience for our users, innovate faster, by owning that part of the stack," says Cowling.

Cowling is originally from Sydney but has lived and worked in San Francisco for the last 12 years. He did his PhD at MIT where he met Dropbox cofounder Drew Houston, who asked him to join the company four years ago.

"There was this opportunity at Dropbox, which at the time was a fairly small company, to build a big storage system," he says. "So that drew me in and I've been there ever since.

"It's grown right? It's grown and it's grown up. When I joined infrastructure was just seven people and it was kind of a small scrappy kind of start-up - with a very successful, compelling product. And we're at the point right now where we have a storage team that I think is putting out the best storage system in the world, so at the very least it's been a pretty rapid transition."

Since its inception in 2007 Dropbox had operated a hybrid architecture. All the business logic, databases and Web services ran from its own data centres, while the bulk storage was kept in the cloud on Amazon's S3.

When the likes of Google, Amazon and Facebook began, there were no cloud services to leverage. Those companies built their own infrastructure and were able to start small and expanding when necessary. Dropbox, on the other hand, would have to build and launch a huge infrastructure setup from scratch.

They call this infrastructure Magic Pocket, a nod to what Cowling calls a "very cheesy" 2009 promotional video (one of the company's first).

 

1  2  3  Next Page 

Sign up for CIO Asia eNewsletters.