(Kevin N. Hume/S.F. Examiner)

BART working to prevent future computer system failures after Saturday’s systemwide shutdown

BART trains sat still last Saturday morning, the familiar screeches absent across the Bay Area. But for days after thousands of riders were stranded, one question lingered:

How did this happen?

At Thursday’s BART Board of Directors regular meeting, staff told directors that the entire system was felled by a single “network switch” failure.

Essentially, a conduit for information between the train control system and equipment failed, a broken link in a chain of communication.

“The type of failure that occurred this weekend is very rare,” said Tamar Allen, assistant general manager of operations at BART.

The last time this kind of failure occurred was 2006, she told directors.

“This weekend’s failure was due to a failure of the network switch itself. It was a component failure,” she explained.

And that may not be an issue that ever hits BART that hard again, due to new efforts underway to prevent this “type and magnitude” of failure, Allen said.

Among them is the installation of a “remote redundant” disaster recovery center, which will be built out within the month, Allen said. That is being paid for with federal funding and was already underway when the network switch error occurred.

At about 2:45 a.m. this “single network switch,” one of many in a complex network, failed. It recycled data and generated a “data spike,” Allen said, rippling throughout BART. BART lost communication between its operation control center and any equipment in the field. Cisco Systems helped BART identify the issue.

The data spike meant that in that single network switch “the number of data packages requiring processing increased from one per millisecond to 54,000 per millisecond,” Allen said. That overwhelmed the network switch.

That network failure also prevented BART from enabling trip advisories online and in apps to warn riders that the system was down.

By about 9 a.m. the switch problem was largely resolved, Allen said. Trains were running systemwide except south of Daly City.

“We brought in all of our engineers who work in communication and engineering,” Allen said. “It was a large effort, and a lot of people contributed” to restoring the system.

All field sites had to be manually rebooted, she said.

Despite what was described as a “herculean” push by staff to get trains back up and running, the BART Board of Directors held staff to task.

“I clearly hear it was a gargantuan effort to get the system back up and running,” said Janice Li, a BART board director who represents San Francisco. Still, she asked BART staff for assurances this would not happen again.

“I heard so many stories,” she said, of riders left in the rain, of riders inconvenienced or stranded.

Allen told Li, “When you’re running large data networks, errors do happen,” but “as you may have been aware, Facebook was down much of yesterday” due to the same problem.” And with the new redundant data center, such problems may be avoided.

“As president of the board I want to recognize that we did let riders down on Saturday,” said BART Board of Directors President Bevan Dufty. He said he was “grateful to all our staff” to resolve the issue, and for the new redundant system that would prevent subsequent failures.

“We have to demonstrate that situations like this will not happen and we will be a reliable system,” Dufty said. Transit

If you find our journalism valuable and relevant, please consider joining our Examiner membership program.
Find out more at www.sfexaminer.com/join/

Just Posted

Union threatens legal action after Police Commission expands use-of-force policy

San Francisco’s police union is pursuing legal action after the Police Commission… Continue reading

Restorative art on the inside and out

Curator Ericka Scott organizes exhibition of works by prisoners

City Attorney seeks to recoup ‘illegal profits’ gained by Walter Wong through city contracts

San Francisco will seek to recover “illegal profits” gained by well-known permit… Continue reading

SF Police Commission votes to expand use-of-force policy

Decision to bypass union negotiations could set stage for litigation

Free Muni for Youth program expansion halted by SFMTA budget crisis

Low- and moderate-income kids can still travel for free

Most Read