SINGAPORE, 5 AUGUST 2010 Singapore's DBS Bank and its IT services provider IBM have completed their investigation into the DBS systems 5 July 2010 systems crash, which deprived bank customers of digital services for some seven hours.
DBS has escaped a fine, but has been censured by the Monetary Authority of Singapore (MAS) which has ordered it to increase its capital buffer against any future so-called operational risk losses by 20 per cent.
This translates to DBS having to set aside an additional SGD 230 million (US$170) in regulatory capital, on a group basis, based on 30 June 2010 figures.
According to the Singapore Business Times newspaper: The MAS also instructed DBS to adopt measures so that it does not overly rely on a single service provider or a single vendor's products and services'.
DBS CEO Piyush Gupta, said: The system outage is of grave concern to us and we acknowledge MAS' censure.
In their statement, DBS and IBM jointly provided a detailed account of events which preceded the outage, and consequent recovery activities and actions.
The investigation found that an IBM field engineer did not follow approved procedures when responding to an alert which indicated instability in a communications link in the storage system which was connected to a mainframe.
The statement said that the engineer did not use the system's approved maintenance interface but instead, used the instructions given by the IBM support centre. He was called back a second time to replace a cable, in consultation with the support centre, and again used the incorrect procedure to check it. As a result, the storage system ceased communicating in order to protect the data.
If the correct procedures had been used, the storage system would have automatically suspended the communications link and the machine would have instructed the engineer to replace the cable and both cards together and maintain redundancy of the system, the joint DBS IBM statement said.
In a separate report, The Business Times stated that IBM has disciplined both the engineer and the support staff and they have been removed from direct customer support activity.
David Gledhill, Managing Director and Group Head of Technology and Operations at DBS said that DBS has taken steps to improve the bank's internal escalation process and the speed and manner in which it reaches out to customers during periods of service disruptions.
The bank is defining specific red alert' scenarios that will automatically trigger group-wide crisis management procedures, including specific actions to be taken for each of the red alert scenarios identified, Gledhill said. DBS is also implementing additional modes of internal alerts and implementing new processes to expedite escalation.
Sign up for CIO Asia eNewsletters.