Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Why the cloud requires a totally different data center

Brandon Butler | Nov. 19, 2013
AWS VP who builds data centers for the biggest cloud provider says scale matters, a lot.

If there's one mind-blowing statistic about Amazon Web Services, it's the company's scale.

The cloud is a nascent technology, but AWS is already a multi-billion-dollar business and its cloud is reportedly five times bigger than its 14 top competitors combined, according to Gartner. Amazon's Simple Storage Service (S3) stores more than a trillion files and processes 1.5 million requests per second. DynamoDB, the AWS-designed NoSQL database, is less than a year old and last month it already had more than 2 trillion input or output requests.

Supplying all those services at that scale requires a lot of hardware. The cloud division is growing fast though, which means that AWS is continually adding more hardware to its data centers.   

How does AWS keep up with all that? The man who directs the strategy behind it, AWS Vice President and Distinguished Engineer James Hamilton, shared insights into this at the company's re:Invent customer conference in Las Vegas last week. In a nutshell, "Scale is the enabler of everything," he says.

Scale helps the company in a number of ways, Hamilton says. For one, when a company does something a lot, it gets really good at it. So, the more servers AWS provisions, the more it learns about best practices in doing so. The same is true for data centers. The company is adding so much capacity so fast that it allows AWS engineers to test ideas quickly to find out what works and what doesn't, Hamilton says.

AWS has optimized its hardware for its specific use cases, he says. AWS has built custom compute, storage and networking servers, which allow the company to customize down to a granular level. Its storage servers are "far denser" than anything on the market and each weighs more than a ton, Hamilton says. Most recently AWS customized its networking gear to create routers and protocol stacks that provision high performance workloads. Hamilton says those investments will drive new networking-based products and services that will be released in the coming year. The idea of creating customized hardware is not unique in the industry - Facebook has created an entire open source project named the Open Compute Project for this process.

AWS even customizes its power consumption processes. The company has negotiated bulk power purchase agreements with suppliers to get the energy needed to power its dozens of data centers across nine regions of the globe. The US-East region, the company's largest, has more than 10 data centers alone, Hamilton says. Not only does AWS buy power in bulk for these data centers, the company has its own sub-stations that store power before it's served to data centers. This allows the company to be able to scale up its offerings as much as it needs to without worrying about power being a limiting factor.


1  2  Next Page 

Sign up for CIO Asia eNewsletters.