That's exactly what happened in October when AWS had an outage (its third major one in two years) following a new hardware installation ("latent memory bug" issue) at one of its northern Virginia data centers. Websites such as Reddit and Imgur were left to do the explaining to their would-be website visitors, while Amazon followed up with credits for its customers and a promise to fix the underlying problem, including an overaggressive traffic throttling policy. Amazon also apologized, writing in part on its AWS support site: "We apologize for the inconvenience and trouble this caused for affected customers. We know how critical our services are to our customers' businesses, and will work hard (and expeditiously) to apply the learning from this event to our services. While we saw that some of the changes that we previously made helped us mitigate some of the impact, we also learned about new failure modes. We will spend many hours over the coming days and weeks improving our understanding of the event and further investing in the resiliency of our services."
Amazon was forced to publicly apologize for another outage of its Elastic Compute Cloud lasting several hours in June that it blamed on power outages (caused by storms), software bugs and rebooting bottlenecks. Amazon's team wrote: "We regret the problems experienced by customers affected by the disruption and, in addition to giving more detail, also wanted to provide information on actions we'll be taking to mitigate these issues in the future."
*LinkedIn passwords free-for-all
LinkedIn, in June, confirmed reports that some of its users' passwords were compromised after reports surfaced that about 6.5 million LinkedIn passwords were compromised and posted online in a Russian hacker forum, in large part because LinkedIn was using a weak hashing algorithm.
The business-oriented social network site quickly updated its security and ensured users who updated their passwords that they'd be in much better shape. Even so, it was hit with a $5 million lawsuit over the breach.
Here's what one LinkedIn VP blogged, in part: "We sincerely apologize for the inconvenience this has caused our members. We take the security of our members very seriously. If you haven't read it already it is worth checking out my earlier blog post today about updating your password and other account security best practices."
*BlueToad victimized by hackers
Digital publishing company BlueToad revealed in September that the unique identifiers of some 1 million Apple iOS devices that hackers leaked were swiped from its servers. CEO Paul DeHart's admission that his company was the hacking victim helped clear suspicion from the FBI, which the Antisec-affiliated hacking group claimed to have taken the UDIDs from. DeHart said in an interview with MSNBC (and he might be the only exec from any of these companies that apologized on camera) that his company did change its code to comply with stricter Apple guidelines earlier this year, but that the hackers got access to information stored via older code.
Sign up for CIO Asia eNewsletters.