The Butterfly Effect refers to the phenomenon where a small localized change in a complex system can have very large effects elsewhere. Nowhere is this more evident than in operating a data center; the steady hum of commerce can cascade into a cacophony of panic within moments. But first, let’s take a step back and see how it happens, and more importantly, how to help prevent it.
The lights are extinguished in a remote, undisclosed, unmarked, fenced-in, security patrolled, grey building extended off the highway in Anywhere, Central Virginia. As electricity courses through the veins of conduit within the building, computer servers flash blue and green lights, processing calculations for logistics systems in Europe or food preparation factory lines in Pennsylvania. The power is distributed through a sophisticated network of copper wire and circuit breakers, engineered to protect the entirety of the system in a procedural and logical manner. The inevitable component failure or improper installation along the various pathways requires contingency planning so the Butterfly Effect does not mix up orders of Chianti and crude oil or grind the pecan pie factory to a halt.
As a part of the Bala Commissioning Team, we are acutely aware of the level of detail required to prevent such a butterfly effect from paralyzing, or even impeding, the continuous operation of a Data Center. Financial institutions, healthcare systems, and technology networks all require constant, uninterrupted activity.
To assuage the managers and owners of these facilities, the Bala Commissioning Team conducts customized commissioning procedures to test and verify the building systems have been correctly designed, installed, and operationally vetted prior to entering full production mode. Systems can be pontificated upon and integrated with the best intentions, but without conducting a real-life test scenario, the risk of figuring out the unknowns during time-sensitive critical moments exponentially increases. Commissioning a Data Center bridges the unknown-unknowns with confidence.
We call it Integrated Systems Testing, or IST for short. This is “Level 5” commissioning where the back-up systems are tested by simulating a facility total loss of utility power (“pull the plug”). We want to prove that all the elements of the system respond as designed and transfer power to either an alternate utility or temporary power generating source.
Level 4 testing, in contrast, is performed at a component level by testing individual equipment to verify its operation.
Our experience testing a multitude of sites has taught us that flawless operation is rarely achieved without testing. During recent Level 5 commissioning of a new Data Center for a major financial institution, our analysis uncovered two circuit breakers that were not set at the correct fault tolerance levels. The fault event halted our test procedures until we traced down the source of the deficiency. If this occurred at any time of the day during a loss of utility power, it would have severely impaired normal operations.
At another technology-focused site we found computer room air conditioning units were not configured with the critical facilities management system and did not return online during a loss of power event. Even further, we found the failure events were not electronically reported to the critical facilities management system to notify personnel of the adverse incident. In this case, it was a latent issue that could have been missed by management if communications to the systems were lost.
IST is an invaluable exercise performed by Bala’s Commissioning Team that simulates a total loss of power event, often in multiple failure scenarios. Without testing the entire Data Center as a fully integrated system, you might as well be shuffling the deck chairs on the Titanic. Bala’s Commissioning Team is here to keep you sailing smoothly and have confidence in the design and unfettered operation of your reliable Data Center.
