Why DR fails [2]

What became clear is that correct maintenance of production systems can actually lead to an INCREASE in the number of DR gaps, since keeping the system working means more change, and more change means more gaps, and so on. Current Enterprise Management Consoles, SRM systems, etc. just don’t help, because the problem they were intended to solve actually conflicts with DR assurance.

The next step we took after understanding the issue at hand was to define requirements for a DR and data protection gap detection tool. We begun by analyzing what could get wrong, figuring that if we could have found “signatures” of gaps, we could build an engine that can constantly search for them. Think of your DR checklist – things you need to check before starting a DR drill – and imagine how many tests you have there – dozens? This sounds about right, and in fact, when we started working with customers it turned out that each could contribute its share. It also happened that the overlap was significantly smaller than we have anticipated, so that the accumulated list grew alarmingly fast to contain thousands of possible gaps. As the magnitude of the problem became clearer, so was the conviction that no team of men or women can check everything manually. Some of the tests could take days to complete for just one server (!) – even when furnished with top-of-the-line monitoring, automation and SRM tools. Imagine how much time it would take to search for hundreds of gaps in a deployment of hundreds or even thousands of DR-protected servers. The analogy that comes to mind is looking for viruses manually with a checklist containing printout of thousands of virus signatures.

The way to find a solution was challenging and rewarding, resulting in our first three patents. I hope to dedicate my next post to some more of our insights regarding DR gaps.

Why DR fails [1]

Have you ever found yourself wondering why, despite all the effort, preparation, concern and constant care, DR systems seem to elude us by not working just when we need them?

In the last 18 years I've been trying over and over to define yet the perfect DR architecture for the organizations I was involved with. While I was fortunate enough to have the satisfaction of building several complex and neat setups that actually worked, solutions just didn't seem to scale to the larger environments. New replication, clustering and automation tools came and went, yet the problem never appeared to get easier.

Over the years, the question kept gnawing at me, with the answer always almost, but not quite, within my reach.

It turned out that there were others occupied with the same question. I met my partners to the quest – the founding team of Continuity Software - back at 2004’s fall, and together we were determined to find a satisfactory answer. It took some time, but finally gave in. The "revelation" - as with many other new ideas - turned out to be quite simple. There is a missing process, or tool-kit, well known in other fields of IT, but missing in DR. Without this tool-kit, no matter how skilled or devoted we are, chances are little to have a working solution for long. Let's explore this idea.

The larger and more complex our IT environment gets, the faster it will change and become prone to disorder. This phenomenon is quite obvious. It demonstrates principles underlying any system, and formulated in many fields of life and science (take for example the Physics field of thermodynamics with the fundamental notion of entropy).

Disorder will not disappear by itself - we have to introduce mechanisms to control it, or, more precisely, to negate its effect. This tends, BTW, to restore order in one place, by creating an even greater degree of disorder elsewhere - again, following "rules" of nature. In various IT areas, the niche is usually filled by monitoring and management tools. These will usually let us identify evolving problems and react by introducing the required changes to put the system back into working order. As expected, it will rarely bring our systems back to any past state, but rather to a newer, possibly more complex one. For example, think of an application that requires more storage space. You add the missing Terabytes. After several days you realize that a related data-warehouse load-process no longer finishes on time (hopefully, you have a monitoring tool tell you that rather than your users...). Obviously, you would not consider removing the added storage space, but rather add more CPUs, more bandwidth, etc., until it works again.

On the other hand, DR environments are sensitive to change; anything we do on production should be faithfully reproduced on the DR site. How can we tell for sure that nothing got forgotten, or worse yet, was done incorrectly? Since DR testing is intimidating, complex and extremely costly, it is rarely exercised frequently enough. The result is that implementation gaps exist, and grow in number constantly.