Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

A data center story for the ages: the fuel bucket brigade

Patrick Thibodeau | June 27, 2013
The effort to save a NYC data center following Hurricane Sandy is remembered in a new film.

There's an area set aside for Peer 1 customers to work on their systems.

As flood waters entered the lobby, Jeffery Burns, a worker in the data center, told of hearing the sound of a waterfall in the elevator shaft that eventually flooded the basement and disalbed the fuel pumping systems.

"Everybody in Manhattan has the same problem, how to relocate infrastructure that - for the last 50 years - has safely lived in basements," said Michael Mazzei, manager of the Peer 1 Broad Street data center.

Despite the damage caused by Hurricane Sandy, data centers won't be leaving Manhattan anytime soon. The island is one of the most networked places in the world.

The Peer 1 data center is in the former International Telephone & Telegraph headquarters building and has multiple network connections that give customers quick access and low latency. It's also a short walk to the waterways.

NYC Mayor Michael Bloomberg this month released a multi-billion dollar, multi-decade plan to protect the city from powerful storms, and the impact of climate change. The plan includes the construction of barriers in low-lying parts of the city.

Bloomberg's plan has broad IT and telecommunications implications.

It seeks a higher level of reliability and service for telecommunication facilities. The specific recommendations include requiring new hospitals "to increase their IT and telecommunications resiliency by installing two independent points-of-entry for telecom and communication to reduce the risk of outages from a single supplier."

But for a business such as Fog Creek, the hurricane exposed its lack of a back-up plan. The company has been working to fix that weakness in the months since.

For instance, Fog Creek has improved its data replication capabilities and is developing an ability to switch operations, in an emergency, to a separate facility. One tool now runs in Amazon's cloud service, though that migration has been planned prior to the storm.

The storm made clear to Fog Creek that it hadn't paid enough attention to the possibilities of bad things, like a massive storm, happening, said Pryor. That was a positive outcome of the experience, "because it did kick us in the butt and get us to fix a lot of things that were broken."

In his office, Mazzei points to the part of the floor he slept on -- next to his phone.

The bucket brigade wasn't the first move by the group. Initially, they tried moving 55-gallon drums up 17 flights of stairs, but the physical exertion proved too much.

Gradually, though, a realistic plan came into place. The team acquired supplies, pumps, hoses and buckets from Home Depot and other places. Organization, rules and efficiency arrived.

 

Previous Page  1  2  3  Next Page 

Sign up for CIO Asia eNewsletters.