At a Computerworld Hong Kong event years ago, I heard a senior executive explain how online retailer Amazon re-engineered their e-commerce process for the Christmas gift-giving season.
Like many retail businesses, Amazon experiences an extraordinary surge in business during this time. But unlike bricks-and-mortar shops, Amazon's business--from order-taking to payment-processing--is data-centric.
The executive said that Amazon's "Christmas rush": a flood of orders requiring a supply-chain that guarantees delivery by December 25, couldn't be replicated. So the e-commerce giant built a test-system which ran in parallel with their online system, and ran EVERY order through it simultaneously to see if it would break.
And they did it two years in a row, he said, before migrating to the new system. Amazon couldn't afford a misstep with their mission-critical business engine. Not during the short timeframe when the data pours in furiously and every last byte needs to be processed, because shops that fail to deliver on Santa's schedule make kids cry, and parents have long memories.
Carat Cheung Ming-nga was shedding tears last week--tears of joy at being named Miss Hong Kong. But Netizens who expected to have a say in the voting shredded their handkerchiefs instead, as TVB's much-anticipated online voting scheme went haywire. The aftermath: apologies from TVB executives, finger-pointing, press-conferences, Internet-forum rants, and hundreds of complaints to the Communications Authority.
What went wrong? The "H-word" (hackers) was immediately trotted out. TVB's general manager for broadcasting Cheong Sin-keung said: "There was an unusual high volume of traffic, and we don't rule out the possibility that hacking activities were involved."
But was it? Websites in Hong Kong like Urbtix and online events like the POPvote straw-poll earlier this year have experienced traffic-spikes that blitzed their servers. The problem is compounded when Netizens, eager for tickets to a Joey Yung concert or the Rugby Sevens, repeatedly hit the "refresh" button, further overwhelming the system. No hackers involved, just a data-stampede.
The problem is scalability, and TVB's strategy was to leverage a cloud-based platform to handle the traffic: Microsoft's Azure. That's a good plan, but as ever, the devil is in the details.
"People seem to assume that cloud gives you unlimited resources in every possible way, and it just doesn't," said Richard Stagg, managing consultant at Hong Kong-based security consultancy Handshake Networking. "More bandwidth, maybe, and more flexibility, but in the end a Web application has as much resources as it has been allocated and those resources can be consumed just like any other Web server."
Also, cloud-based computing resources are first and foremost computers: when they're overloaded they may require rebooting, and reboots are never instant. While cloud is a more scalable platform, as Stagg noted, it requires allocation of resources and cost-benefit analysis--all subject to TCO and ROI scrutiny. Sometimes it's worth it: Amazon valued their e-commerce operation so highly they ate two years of operating costs for a test-system to ensure it would handle peak loads under time-critical pressure.
Sign up for CIO Asia eNewsletters.