By matt buchanan

Inside AT&T's National Disaster Recovery Batcave: Who AT&T Calls When the Death Star Explodes

When the World Trade Center collapsed, it took out a critical AT&T switch, crippling service. It was restored in 52 hours—including the time to drive a caravan of eighteen-wheelers from Atlanta to a lot in Jersey City.

***

It's hot, and muggy, like it usually is in Georgia at the end of July. There's no AC in this warehouse, a concrete desert with a tin roof, lit by strips of undying fluorescent lights and streaks of the sun flooding in from the open bay doors in the back. A single industrial-sized fan is blowing, almost like someone's idea of a practical joke. It's a vast industrial space that feels utterly empty, even with dozens of 18 wheelers lined up, a convoy waiting for a calamity. The only signs that humans work here are a basketball hoop and a climbing rope. I was hoping for a more Batcave-y Batcave.

This nondescript warehouse, in an even more nondescript town outside of Atlanta, is home to AT&T's National Disaster Recovery program, one of six depots scattered around the country. When an AT&T central office or other piece of critical networking infrastructure, is wiped off the face of the earth, NDR is what makes shit work again.

***

The World Trade Center came down on top of a node in tower number two and took out a switch for voice calls in the adjacent building. The 52 hours it took NDR to restore service included not just the time it took to drive a caravan of eighteen-wheelers straight up the East Coast, but picking up stranded workers all along the way. (The FAA had grounded all flights into NY.) The trailer city became AT&T's de facto office in New York for three months. That's what NDR is, effectively—AT&T on lots and lots of wheels that can set up anywhere and assimilate any function of a "smoking hole" office. It's like the Super Star Destroyer from Empire, after the Death Star is blown up.

I guess the simplest way to think about NDR is in terms of the fleet: over 300 trailers nationwide, loaded with networking equipment. The warehouse I'm standing in houses around 40 of them. When AT&T's Global Networking Operations Center, located in New Jersey of all places, declares that a disaster has ensued—an office destroyed, a switch crushed or something else that majorly crimps the network—a convoy is dispatched. On average, it takes 20-40 trailers (think eighteen-wheelers) and about 96 hours to replicate an office, depending on the kind of service they're trying to restore.

The guts of AT&T, vis-a-vis the microcosm of a tractor-trailer, are what you'd expect, almost mundane: a lot of cables and switches and wires. Trailer after trailer, the hallways, lit by cold fluorescent lights, are straight out of your more claustrophobic nightmares of Area 51. But disappointingly, without dripping goo. They're almost like giant routers stretched across a hallway. Super duper routers, really, since the newer trucks have the capacity of an OC-768 line (38.486016 gigabits a second), or 80,000 voice circuits. And in the back, man-sized modular batteries that last for up to a decade.

And then there's the "Home Depot on wheels." One of the things that AT&T's learned about disasters like Katrina since starting NDR in 1991 is that if you need something, bring it with you—with thousands of crazy people running around, it's probably not going to be there. The trailer is a full workshop, with enough saws, hammers, power tools and materials to build a small fortress or seven. It smells like a toolshed should, the dirt and grime and plywood a nicely organic contrast to the yellow plastic and blue cables lining the surrounding networking trucks.

When an NDR team lands in the middle of a shitstorm, first it looks for somewhere to setup. It needs space, and if possible, access to fiber. For the WTC, Manhattan wasn't an option. So it investigated three vacant lots in New Jersey, all located near troves of "dark fiber," masses of inactive fiber that's already been laid down, making it easy to jack into the network. (Preferably, it can just plunk down next to the dead office, like in Galveston, post-Rita, where it spliced in via manholes.) Trailers are then set and leveled off. Phone lines established; then power is jacked in and grounded; networking and optical lines between the trailers are hooked up; and finally tech people start flipping switches to start rerouting everything where it needs to go. A weekly data backup is made for each office—Lucent merges the tape with configuration data to establish all the circuits correctly. All of this is what happens in the course of 4-7 days.

***

We're back in the conference room, where the morbid business of disaster planning takes place. A map of the US on the wall is filled with red pushpins, looking like it has a weird case of the measles. It's where they've held training exercises, which happen once every three months, since actual disasters don't happen quite that often. A weatherman appears on the TV hanging overhead, blathering about the latest on Tropical Storm Bonnie, and where in the Gulf it might head next. My AT&T escorts look up, briefly fixated on the map he's gesticulating toward. He might be telling them where they're headed next.