Disaster recovery environments as potential DevOps workspaces
Some companies find that, as they further embrace DevOps on a deeper level, their developers will constantly ask for access to the hot spare disaster recovery environments to mitigate that “single laptop or desktop” constraint. Generally, midsize and larger organizations have significant investments in backup infrastructures that are almost exactly carbon copies of the environments that are running critical production workloads and are at standby in case those workloads need to be ported over in the event of a service disruption or a disaster.
Today, they are mostly standing idle except for a few recovery exercises each year. But think about it: That’s a lot of unused capacity sitting there waiting for an event that may happen for just a few hours a year. Is there a way to take advantage of virtualization technologies, storage area networking and software-defined networking so that the hot spare disaster recovery capacity can be used as a DevOps “workspace” for testing and planning while also remaining available for its original purpose — to accept failed-over workloads?
The answer is yes. Hypervisors and virtual machines can be spun up and down in a matter of seconds, and software-defined networks can be rerouted and shunted to different connections and endpoints via scripts in an automated fashion. In fact, in most cases the data is already accessible to the environment, so regular application and operations testing can actually work with real data without long copying times. Could you ask for a better real world performance testing environment than to work in essentially a carbon copy of your production infrastructure?
In the event of a need for failover, you can have your monitoring system fire those scripts, change the storage endpoints and shut down the virtual machines — and then you have your hot spares back. At the conclusion of the “disaster,” you can manually restore services by reverting all of the changes those automated scripts made. PowerShell is a great way to do this in Windows, Hyper-V and VMware environments, and Bash scripts are decent for Xen and other hypervisors, too. Of course, Puppet and Chef and Ravello can help out with this sort of orchestration as well.
The idea here is get some of that unused capacity doing something useful while also not completely missing the purpose of its existence in the first place. Developers need access to that big iron to do actual testing and teasing out performance problems at higher capacities and loads than their development machines alone can support. Having hot standby infrastructure doing nothing but being hot and standing by is perhaps the polar opposite of embracing DevOps; by reimagining this sort of application, it’s possible to have your cake and eat it, too, if you will.
Sign up for CIO Asia eNewsletters.