Muni’s computerized automatic train-control system is getting an upgrade.
And it’s about time, too, officials say. Muni’s mainframe harkens back to the mid-1970s — making its contemporaries Atari’s “Pong” and floppy disks.
“We’re taking advantage, if you will, of more and more technology,” said John Haley, director of transit at the San Francisco Municipal Transportation Agency.
The stakes are high, as 1998’s Muni Meltdown hasn’t been forgotten. That’s when a computer upgrade went horribly wrong and ground more than half of Muni’s light-rail fleet to a halt.
Muni riders were furious. The San Francisco Examiner’s headline after the delays read “Muni Hell on Wheels.”
For better or worse, the Muni Meltdown made the SFMTA skittish to upgrade its computerized train-control system all at once, Haley said. Instead, it has done it piece by piece, and now the agency is at a turning point in making some of those upgrades.
Muni is currently in the middle of overhauling its train-control systems. Some upgrades are complete, while others are set to finish by November.
The first of the upgrades required rewriting the software that controls trains on the railway, allowing trains to automatically berth next to one another, when two trains appear at a station simultaneously.
The newest upgrades will mothball the myriad banks of clicking copper to enable the newer relay system, only 2 feet wide, to monitor the tracks on its own.
As Haley led The Examiner on a tour of the agency’s old mainframe, he said the transition will be much smoother than in 1998.
In a dank walkway tucked away inside the Van Ness Avenue Muni station is the mainframe that controls Muni trains underground.
Relay racks extend down a cramped hallway nearly 15 feet deep, where dozens of 4-inch copper filaments clatter up and down like teeth. The speedy “click, clack” sound signifies a smooth running Muni light rail automatic control system, which commands the J-Church, K-Ingleside,
L-Taraval, M-Ocean View, N-Judah and T-Third Street trains through the tunnels. The clapping beat signifies the proper alignment of the railway’s 83 axle (controls) and numerous other track switches.
“We have the chance about 2,000 times a day to screw up,” said Haley, referring to the number of entries light-rail vehicles make into tunnels every day.
Rail cars are under operator control outside the tunnels. To ensure proper transition to full computer control, drivers must guide their trains with precision.
If the train enters a tunnel at the wrong speed, Haley said, “The train slows down to under 10 miles an hour and it limps through the subway … everything behind it gets delayed.”
When this happens, or a track switch is damaged, a green relay cable running the length of the underground track is tripped. This signals the maintenance crew, which then checks each ticking copper relay for those with an off beat, or ones that have stopped altogether.
Many more systems must be checked: Above the clattering copper run hundreds of wires in and out of electric ports, resembling the back of a television multiplied several times.
The relays “tell us what they see,” third-year maintenance crew member Arvin Geronimo said.
Crews maintain and inspect this substation on many nights in any given week, as the older relays are also more susceptible to malfunction.
“Last year, we had water up to here” Haley said, gesturing with his hand to his waist. The relay system is under a city drain, he said. A yellow tarp is strung at the roof of the relay room, while sandbags, a bucket and more tarps are tucked away in the corners.
“Because you have all five lines going through the subway,” Haley said, “this has turned out to be the biggest area of vulnerability.”
Jim Kelly, deputy director of transit, said that just like trying to service an old MS-DOS computer, the components and experts for the aging relays faded away with time.
“These things are hard to replace,” Kelly said. “You can’t just go to Relays R Us.”
The old system runs simultaneously with the new. A smaller, 2-foot-wide version of the relay racks stands next to a metal cabinet with more recognizable modern computer components. Just around the corner, a simple black laptop is tucked into a cupboard, a control system for an essential upgrade.
“By going to the new system, reliability is much higher,” Kelly said. But this slow upgrade is just the start of the SFMTA’s biggest challenge. As hundreds of new light-rail vehicles will soon join the fleet, Muni’s main train-control computer system must be upgraded as well. v Kelly says they’re not sure yet what that new “brain” of Muni will look like.
The crew that maintains and inspects the relays every night said it is hopeful for the new system — but it also respects the old one.
“I trust it to run another 20 years,” said Hoa Huynh, a 20-year maintenance crew member. But, he said, “It’s slow. Very slow.”