We at AIS have definitely been paying close attention to how our “brothers in arms” at other data centers on the East Coast have fared during and after Hurricane Sandy.
The 451 Research article below summarizes the status of many high-profile facilities.
We send our best wishes to those who have experienced setbacks large and small.
Hang in there, folks, and please let us know if AIS can help you bounce back even stronger than before the storm.
Brian Wood, VP Marketing
Sandy Update: Datacenters Breathe Easier as Power Is Restored
The New York metro area was hit hard by Sandy, a super storm cell that produced hurricane-grade wind and flooding at its center, and snowstorms on its outskirts. Flash flooding, high tides and rain spelled disaster for datacenters in Manhattan and downtown, many of which housed generator tanks and pumps in sub-basements.
T1R wrote about the immediate effects on datacenters in the area, with more than a few locations, most housing several multi-tenant datacenter (MTDC) providers, having outages that lasted minutes, hours and even days. Today, most providers are back on utility power and inviting customers back in to assess the damage and return to normal operations.
Equinix – Equinix reports that all New York and New Jersey datacenter sites are back on utility power after several days on generator power. The company further reports that its fuel supply never reached less than 50% of capacity at any site with daily and overnight deliveries. Equinix experienced an outage for one hour and 15 minutes during switchover at NY9 due to a failed generator. It also reported that its Philadelphia and Washington DC locations remained on utility power throughout. All facilities are open to customers.
Internap – Internap’s last update reported that all customers should be up and running as normal at both 75 Broad and 111 8th. The company had outages at both locations when 111 8th experienced malfunctions of the building-fed fuel system and when its fuel supplies were depleted in its secondary tanks. The company reports that a generator was delivered on October 31 with 40 hours of fuel on-site.
Verizon – Verizon headquarters at 140 West Street in Manhattan suffered extensive flooding and as a result, severe damage to its switching facility. Water flooded the subterranean cable vault, as well as the company’s five-level basement and its lobby. Emergency generators were obtained with critical cables run through a second-story window to restore functionality. Verizon continues to work on resolving issues, but has no time estimate as to when all services affected (wireless, Internet connectivity and landlines) will be restored.
Datagram – It was five full days before Datagram at 33 Whitehall could secure a generator on-site to restore power to its datacenter, but only after removing water from its building and basement where it affected electrical components. Currently, the company is operating on generator power and has no information as to when utility power will be restored.
111 8th – 111 8th is home to several MTDC providers including Internap, Telx, Equinix and Zayo Group. Internap reports that the facility is running on utility power as normal. Zayo Group reported that cooling systems had to be shut down due to generator issues. The company reported temperatures of over 100 degrees last week, but have since brought cooling systems back online and stabilized the temperatures. The datacenter is running as normal, and all customers are operational.
75 Broad – 75 Broad experienced several issues due to flooding of the basement that housed both electrical components and pumps for the diesel generators. Both Internap and PEER 1 struggled to maintain power during the aftermath. PEER 1 partnered with customers Squarespace and Fog Creek Software to haul fuel up 17 flights of stairs to maintain operations. The company reported that it was able to sustain uninterrupted service until roll-up generators were brought on-site. The company is not aware as to when utility power will be restored.
DuPont Fabros Technology – DuPont Fabros reported that its Piscataway location was on generator power during utility power outages. The company has since been reconnected to utility power, having operated without interruption during and after the storm. DuPont Fabros reports that the facility sustained three utility outages and operated for about 100 hours on generator power with no issues. This is the longest power outage experienced at the facility.
Webair – Webair’s flagship datacenter in Garden City, New York, had no major damage and performed without service interruption during and after Sandy. Webair activated its emergency response program (ERP) prior to Sandy, which included testing and servicing of generators over the weekend, confirmed fuel levels were optimal (10,000 gallons on-site) and ensured the facility was staffed with multiple system administrators, network administrators and datacenter engineers.
Make it hurricane-proof
There continues to be controversy surrounding the building of datacenters in hurricane-prone markets, including those on or near the Gulf and East Coasts. Unfortunately, demand drives location and areas like New York City and Houston remain popular datacenter markets. There are definitely some measures that datacenter owners can implement to prevent outages while weathering the storm. While many people look strictly at power supply, they often don’t consider the potential implications of flooding, excess winds and loss of city water supply.
Utility power loss remains the top source of outages during hurricanes. Providers must ensure that backup systems are functioning correctly to ensure uninterrupted service. In the case of Sandy, the utility company continues to restore power 14 days after the storm. Some datacenters are still relying on generators and fuel deliveries to operate. Some best practices include locating the datacenter within the same grid as a hospital.
Datacenter providers should evaluate the location including whether or not the land is in one of the many flood plains. Providers in Manhattan found themselves at high risk because of their location in Manhattan’s Zone A, or most prone, flood zone. The buildings were equipped with generator and electrical systems in basements that flooded immediately as the area went underwater, causing extensive failures.
Facilities relying on chilled water cooling systems may find themselves in a bind if they lose municipal utility water feed. Datacenters relying on chilled water should also have reserve water holding tanks, or a backup system that does not require water or outside air cooling, because hurricanes are associated with warmer temperatures.
While datacenter operators should be prepared year round for any type of event that may result in outage, those located in hurricane-prone areas should include additional preparation prior to hurricane season and again if they are predicted to be in the path of the hurricane.
Get ready before hurricane season
Hurricane season runs from June through November, with the most activity occurring in August and September. The most hurricane-prone areas include the Gulf Coast and East Coast up through North Carolina/Virginia, although, as Irene and Sandy have proven, states further north should not feel as though they are excluded. Datacenters in these areas should be proactive in hurricane preparedness and should share this information. Some may choose (wisely) to involve customers in the process and in any performed drills. This is a good opportunity to address customer concerns.
Providers should ensure that emergency response procedures (ERP) are in place and tied to case-specific emergency procedures (utility outage, generator failure, UPS failure, chiller failure, etc.). Providers should evaluate the need for updates or changes and revise if needed.
The datacenter should conduct periodic full operational tests to ensure that not only are the generators working, but that they will work correctly with other systems and carry the full load in the event of a utility power outage. This process is called full load testing and involves more than the typical turning on of generators to ‘exercise the generators’ without a live load. Semi-annual testing of the entire system, including removing utility power to the datacenter, should happen to ensure that all of the equipment operates together as designed.
Additionally, all employees should conduct regular drills: fire drills, power outage drills, flood drills, cooling failure drills. Operational staff and engineers should be knowledgeable not only about how equipment should operate under normal circumstances, but also on how systems should respond under abnormal circumstances. These drills should be part of a comprehensive maintenance plan.
Providers should maintain an emergency contact list, ensure it is up to date and make sure it outlines exceptions and/or temporary changes. Providers should identify and add secondary contact methods, and ensure contact list for vendors is up to date and includes any exceptions, changes and secondary contact methods.
Spare parts become increasingly important when roads are closed following the storm. Providers should populate on-hand spare parts before hurricane season, as well as conduct an inventory and identify storage locations. Providers should start with critical components, add vendor-recommended spares and ensure maintenance and operational history is evaluated to identify trends of regular lifespan to complete supply. Maintaining an immediate replacement policy on any parts used during hurricane season can become critical in the difference between uptime and downtime.
The storm is on the way
While it’s critical to test emergency systems year round, any datacenter finding itself in the potential path of the storm should check once more. Providers should conduct drills. There’s that word again, but seriously, employees should be put through the paces, the same as the equipment. Ensure that personnel are familiar with the contents of the ERP and the workflow; review any changes made during the last update and changes occurring since the last update; clarify any questions or confusion from operations staff; and identify high-probability failures and conduct practice drills prior to the storm’s arrival.
Now that the systems are tested and employees know what to do when the power goes out, it’s time to do the same type of preparations you might do for your home. Providers should top off fuel for generators and ensure that they have immediate access to regular fuel deliveries in the event of a power failure. Providers should also stock up on supplies. If people can’t get out, neither can DC providers. This list includes food, water and sanitation supplies for all people who will be on-site for the duration of the storm.
Finally, vendors and contractors should be alerted to the potential for problems during the storm. If you don’t have the personnel on staff to bring the facility back in the event of failure, the proper vendors and contractors should be contacted in advance to ensure availability and priority.
After seeing the aftermath of hurricanes in New York and Houston during the last four years, the earthquake in Japan, the rolling blackouts in Texas during Super Bowl weekend 2011, and flooding in Brisbane, Australia, T1R believes that many providers took appropriate action in planning for Sandy. The Northeastern coast is not known for being a high-risk hurricane area, but providers responded well to the crisis at hand through improvisation and ample communication before, during and after the storm.
T1R believes that despite outages, providers did well in communicating issues and plans of action to customers through multiple outlets. Some were able to restore power through extreme methods (carrying fuel in buckets up 17 floors) and others were able to obtain roll-up generators. Still others worked with competitors to install temporary equipment in alternate facilities. T1R continues to believe that quality customer experience, especially in the event of a crisis, will retain current and attract new customers in the future. T1R also believes that having weathered this storm, many providers in the area will be looking to make changes to the location of electrical and generator components, and customers will get a little more serious about disaster-recovery sites away from the coast.