Why DR tests fail?

Because people cannot be 100 percent sure if their DR setup which often cost millions will actually work. Tests are the only definitive way to verify DR. I am not trying to be arrogant or cute here. As an IT professional with a dozen years of experience, I know that sometimes I do not know well enough and I am not alone in this mud. Have you ever met a DBA who does not know exactly who and which applications are using his/her database? Have you ever met a network manager who does not know surely who are using his/her network services? For those managers who think they know, can they be sure there will not be applications running under users ids that had not been assigned for the current purpose? True the majority of the managers do their due diligence. They monitor regularly. But unless people have been doing so long enough and been on their jobs long enough, they will not be able to completely rull out small odds. There will be jobs which run once a year (or once every 5 years.) Those seldom run jobs are often critical ones. They will be important corporate tax schedules, regulatory reports or court summons. The analogy can be found not only in IT business but almost everywhere. I used to live in an old house in Brooklyn for several years. There were many pipes and cables behind the sheet rocks that I had never been able to figure out which ones were doing what.

For IT business, a DR test is a good opportunity to straighten out the proper relationships, to dig out the hidden dirt that nobody has had any idea of. For large IT operations where there are thousands of servers and hundreds of peoples involved, there will always be dark stop that has never been brought under sun light. For this reason, a DR test failure is never a failure. It is simple a new guidance, a hit of what we should do next. A failure in the real sense may happen only when we assume we know it all and we stop trying.

Someone suggests that we do not use the word “test” but use “exercise” instead. That way we will avoid being infested by the word of “failure.” If changing a wording is all it takes to make us feel good, let us just do so today. Hopefully, we will not need to toil too many weekends on DR exercises before we get rid of all the kinks and achieve a total DR victory.

Note: After Saturday’s DR testing, I did go to Shawnee Mountain to ski for day. The snow had been the best for this year.