Given the dire warnings about climate change, some business and IT people are pondering this question: How should datacentre managers handle the crop of 100- and even 500-year storms, coastal flooding and other ecological disasters that climatologists predict are heading our way?
Some experts suggest that managers of mission-critical IT centres simply need to harden existing facilities, other observers say they need to move the cenres to higher ground and a third group says both strategies are needed.
But one thing is certain, experts say: Few IT organisations -- even those that suffered or narrowly escaped damage from recent major storms -- are thinking long-term. Most IT leaders are, if anything, taking the path of least resistance and expense.
For instance, the datacentre response to Superstorm Sandy, on the East Coast at least, "is nothing more than hardening existing centres," says Peter Sacco, founder and president of PTS Data Center Solutions, a data center design and consultancy firm in Franklin Lakes, N.J. On the other hand, he says, because computers are networked, that deemphasizes "the importance of any single data center."
Internap, a colocation vendor, is strengthening its most at-risk facilities, including the building at 75 Broad Street in downtown Manhattan that flooded after Sandy hit. During the storm, fuel pumps shut down and Internap switched to a 1,200-gallon reserve fuel tank on a higher floor to keep servers running.
"Nobody expected Sandy to become as catastrophic as it was," says Steve Orchard, Internap's senior vice president of development and operations. "With Hurricane Irene the previous year, we're seeing a trend that's a little alarming."
The company has announced it's building a new datacentre in Secaucus, N.J. -- outside the flood plain. "We take climate change very seriously, and it does factor into our new site selection," Orchard says.
Hurricanes Katrina and Rita, followed by Gustav and Ike (2008) and Isaac (2012), slammed into the Gulf Coast with such ferocity that IT executives at Entergy, a $10 billion electrical power company with 14,000 employees, abandoned the idea of a single data center in New Orleans and went back to the drawing board.
Before Katrina hit in 2005, the power company, which provides both nuclear and fossil fuel electricity to 2.8 million customers, had its corporate headquarters in New Orleans and one data center in Gretna, La., just west of downtown New Orleans across the Mississippi River.
"We knew the data center was in the storm's way, and we made a decision after that event to move the data center because we were holding our breath," says Jill Israel, Entergy's CIO. "We didn't have flooding in the immediate area of our data center," she explains. "But there was no power, our lines were down and we had to run on our generator and keep topping it off. "
By winter 2006, the company decided to create two mirror-image data centers in Jackson, Miss., and Little Rock, Ark. In Little Rock, the company retrofitted an old library with sturdy brick walls, moving hardware and critical applications from New Orleans, piece-by-piece, to the backup facility by 2008.
Finally, by 2010, Entergy had completed its brand-new $30 million Jackson, Miss., data center. The company load-balances several systems including email between the two centers, Israel says. "Moving applications from New Orleans involved quite a choreography plan. Subsequent to Katrina we've had [major] storm events, including ice storms in Little Rock. But I no longer have to hold my breath," she says.
The company holds drills for hurricanes and storms every year "to get better and better at responding," Israel says. "One of the things we quickly recognized was how effective a dispersed workforce can be. Our employees can do a lot more things from remote locations and that has served us very, very well."
Not everyone seems to have absorbed the "take heed" message. IT shops in both Europe and America's Northeast seem to cling to the idea that superstorms are non-repeatable freaks of nature. In some cases, even among those affected by major storms, vigilance plays a chess game with artful forgetfulness as managers choose other IT priorities.
Story continues on next page
In Florida, replication is key
Flooding, high winds and hurricanes are yearly occurrences in Florida, and data center backup has become key to local government continuing to function during disasters. Longboat Key is a barrier island with a winter population of 7,000 people that swells to 22,000 in the summer. Located on the west coast between Tampa and Fort Myers, the town decided to replicate critical data services in a secure mainland facility in 2004.
"Florida was just inundated with hurricanes that year, so everyone fortified their disaster recovery plans and efforts," says Kathi Pletzke, Longboat Key's IT director. "We had to assure that if a disaster were to occur, such as a hurricane, we'd need to set up shop fairly quickly and start serving the needs of citizens."
Longboat Key falls geographically into both Sarasota and Manatee counties. Tom Warner, Sarasota County's disaster recovery administrator, began conversations in 2004 with IT leaders in several western counties throughout the state. The result: Warner co-founded the Florida Technology Disaster Recovery Consortium, with Kevin Kryzda, CIO of Martin County, to help one another with replication throughout the region.
"The thought process was: How can we help each other because we are in Hurricane Alley, and what can we put in place long before hurricane season starts to ensure critical applications can be safe?" Warner explains. They looked at data centers to see which were hardened, which category of hurricane they could withstand and if they had additional floor space to house the counties' servers in their location.
With Sarasota County in the lead, the group eventually brokered a deal with Verizon to secure rack and server space for several counties in one of Verizon's switching centers, a secure brick building with no windows located near a major highway and around 10 miles inland from Longboat Key.
The town has replicated its servers at Suncoast Technology Center -- the former Verizon switching center -- and "though we don't have everything on it, as budgets allow, we've replicated the critical applications, including a mirror image of our production server, payroll, general ledger and public safety data," Pletzke says.
Because there are a lot of elderly people in Florida, "we wanted to replicate our website in [adjoining] Martin County so that in the event of disaster, people could check up on their parents in Sarasota County and see that they are well," Warner says. The consortium has also boosted cooperation among many counties to ensure that IT managers can get emergency access to guest servers and generators to keep computers running smoothly.
-- Arielle Emmett
Ignorance isn't bliss
"What was perceived as a safe area before may not be now," says Rakesh Kumar, a Gartner vice president who specializes in data center and infrastructure issues. In Europe, especially, he cites freezing temperatures, coastal flooding and other unpredictable weather events. In Asia, tsunamis are a concern. "Until we have a major data outage, though, most clients are not calculating for risk or change; they're turning a blind eye to it," Kumar says.
Many of his European and U.S. clients praise the idea of doing thorough risk assessments and thinking proactively, long-term, he says. At least in theory.
But even now, six months after Sandy, most East Coast-based companies aren't being proactive about repositioning their data centers, experts say. "They're expanding in the same locations; they're not even thinking about moving," says Neil Sheehan, a data center architect and principal of Chicago-based Sheehan Partners Ltd.
In fact, he says, "we are looking for expansion for our clients in New Jersey right near the coast, [near] sites that flooded." Sheehan says with proper surveys of 500-year-flood levels, data center architects can determine the ideal height of first floors, so that flooding, if it occurs, happens in parking lots and not in data centers.
Some are paying more than lip service. In downtown Manhattan, at 140 West Street, a Verizon switching center felt the full force of Sandy's stormwater. Five sub-basement levels, including a Verizon cable vault, went underwater. Technicians struggled to mount emergency generators and pump water out through elevator shafts.
Since then, Verizon has had to extract 150 tons of damaged copper cable from lower Manhattan streets, its central office, headquarters and customer sites, replacing virtually all of it with weather-proof fiber cable protected in conduit. "If you take a fiber optic cable and lay it in your bathtub it probably will still work; fiber is submersible," says Chris Kimm, vice president of global customer assurance for Verizon Enterprise Solutions.
By replacing electrical infrastructure and bringing it up to higher floors, the carrier was able to get its own central office back online in about a week.
Some customers are following suit -- moving critical infrastructure to floors that once housed rentable office space. And some are deploying new services, Kimm says, including mobile wireless, wireless IP and cloud computing solutions to allow their employees to work remotely. Others are rerouting their telecom and data networks. "We even have some buildings that landlords have to redo, converting them from business locations to residences and deploying wireless services."
Even given all the damage, however, Verizon isn't considering moving its switching centers to a less flood-prone location. Instead, Kimm says, "We're armoring the buildings; we've done an evaluation of what all the risks are," he says. "We haven't gotten final solutions, but the start of the next hurricane season [is a few months off]; you've got until mid-summer before you have a significant risk of a future event."
Lessons learned?
The impact of climate change and storms like Katrina and Sandy remains difficult to calculate. Not even climatologists can predict the frequency of extreme weather events as ocean levels and temperatures rise. But in the U.S., locations such as Manhattan, Long Island, New Jersey, Miami, Virginia Beach, Boston, Washington, D.C., and even Seattle and San Diego are expected to see increased coastal flooding.
"I think it's absolutely compelling to look at the impact of recent storms, and also to look at statistics that show there have been more natural [severe] weather events, whether that's related to global climate shifts or other factors," says Jim Grogan, a business continuity and resilience specialist at 451 Research.
"Every single event, though, leaves lessons to be learned," he says. "Lessons come from the stories of the creative and innovative things data center operators did to keep their centers" going even in the worst conditions.
Though Sandy may have been a wake-up call for major data centers in the New York area to take some steps to harden facilities, it remains unclear how many will act on a longer-term solution -- moving out of the city entirely, for example, or developing redundant and geographically separate facilities, or opting for third-party disaster recovery and cloud solutions.
New thinking is happening among some of the larger IT organizations. "We're definitely seeing companies looking at alternative locations for data center operations," says 451's Grogan. "We have spoken to multitenant data center operators in Atlanta, Virginia and other locations, and we're seeing an increase of interest from customers in the Northeast."
Many businesses that can afford a disaster recovery solution are looking now toward secured multitenant providers outside major cities. Cervalis, for example, a company with more than 200 large corporate clients including global banks and software and testing companies, has hundreds of thousands of square feet of multitenant space in upstate New York, Passaic County, N.J., and Fairfield County, Conn., all safely tucked away from floodplains.
The company acts as a primary or secondary data center, providing secured cabinets and redundant power, fully loaded (20 to 40 servers per cabinet), for about $1,500 per month per cabinet. That price is still cheaper for companies than building a redundant data facility on their own, insists Zack Margolis, a Cervalis vice president of sales, marketing and business development.
But not all businesses can afford this level of protection. Small businesses might realize the importance of data backups, but most will have to avail themselves of less expensive forms of resilience, such as backup tapes mounted at small disaster recovery facilities or virtualized clouds.
Moreover, many businesses still cling to major cities as the center of the universe. "Downtown Manhattan will still be in high demand because it is an interconnection hub to the United States," says 451's Michael Levy. Further, financial companies still want to "touch their data" and have it near the center of trading action without fear of latency, even though fiber-connected facilities outside the city can do the same job.
Overall, Gartner's Kumar insists, too few IT leaders are taking the signs of weather and climate change seriously enough. "We have had cases of coastal [flooding] where climate change has become an issue," he says. "In London and Germany, the winters seem to be getting slightly worse; we've had cases of component failure -- small bits of electrical equipment freezing up."
Despite all this, most IT managers still aren't willing to make proactive risk assessments to avert disasters. "When I mention risk assessments, they say, 'Rakesh, great point. We'll get engineering to complete a report.' But two months later, it's still at the bottom of the 'to-do' pile."
How to keep going in the next freak storm
You can do many things to plan ahead of a storm. Here are the top picks from those who learned the hard way during Superstorm Sandy.
Secure fuel tanks and pumps. In Manhattan, most downtown data center outages occurred due to ruptured fuel pumps, tanks and shorted electrical wiring. At Datagram -- a private co-location company providing managed services to Gawker, Buzzfeed and The Huffington Post, among others -- torrents of water in the basement blew off stainless steel doors, rupturing a diesel fuel tank. "Our generators were operational, but we had pumps submerged under saltwater, which caused a failure of our fuel systems," says Alex Reppen, Datagram's CEO. The lesson: Install multiple fuel tanks and encase them in concrete so they can't be damaged in a storm surge.
Check emergency wiring, hoses, connectors and back-up circuits. At Peer1 Hosting, with 21 data centers around the country, human bucket brigades brought needed diesel fuel up to a rooftop generator as multiple shifts of people worked overnight to keep a Manhattan data center online. However, "we had a heat issue and lost a portion of our data center for a day because we couldn't [reduce] the heat during the power outage," explains Ted Smith, senior vice president of Peer1 Hosting. Lessons: Review backup power plans, check UPS batteries and make sure there are enough emergency wiring, hoses, connectors and fuel to run backup equipment and generators for prolonged periods.
Get creative in the basement. When basement floors flood, fuel tanks can float, but they also do damage to pipes and fuel pumps. That was a big problem for Internap during Sandy, and New York City regulations require that fuel tanks remain in city basements. "The challenge," says Internap's Steve Orchard, "is that fuel tanks may be underground, somewhat buried, or sitting above ground in basements on stands depending on when the building was built." Lessons: Make tanks submersible by strapping them into place to structural steel so they won't float, Orchard says. "We're also looking into submersible pumps sitting inside the fuel tank along with placing a redundant pump on the mezzanine level." For added safety, Internap could also install a basement pump at a height above six feet.
Consider triple power backup. New York's CitiServ data center in Brooklyn -- a two-building project that is part of Mayor Michael Bloomberg's efforts to consolidate 55 city data centers into one -- stayed up throughout Sandy while managing the emergency 911 and 311 services. "We placed the facility in Brooklyn outside the hurricane flood zone, and we had a triple power backup strategy," which includes utility power, emergency generators and battery, says Rahul N. Merchant, New York's chief information and innovation officer. At the height of the storm, lights flickered briefly, but generator power kicked in seamlessly. A tank holding 200,000 gallons of diesel located on-site at the Brooklyn facility -- and topped off regularly -- powered everything for 45.5 hours before regular utility power was restored.
Re-patch and restore. In addition, the city's own fiber optic network, CitiNet, was used to restore 311 service when Verizon's cable vaults flooded and failed at downtown Manhattan nodes. "We were able to re-patch the 311 network by using our own fiber backbone and connecting it [directly] into the Verizon Central Office in Brooklyn," Merchant says.
Use sandbags. The bullet just missed when Sandy hammered at QTS's data center in Jersey City, N.J. "We were impacted by rising waters and high winds, and the water pushed up right to the edges of our building," says Brian Johnston, QTS's chief technology officer. However, the building was storm-proofed; the actual data center had been located on the 16th through 20th floors, with all power generation and logic on the same level except for electrical vaults and lines coming from the streets. "In the event we lose the vaults, everything we need transfers to UPS; and our fuel oil in 1,000-gallon fuel tanks along with electric pumps in the garage parking lot are also water-proofed," Johnston says. Though QTS temporarily lost utility power, backup generators took over, and no basement flooding occurred. Lesson: "Our skilled staff redirected waters with sand bags. We had wraps around any openings in the building and our parking center, so water couldn't come into parking garage or flood our vaults." Future plans: more flood blockers -- borrowed from hurricane protocols in Miami -- and training for IT staff in emergency procedures.
