Tech in the tunnels: Muni train control system gets biggest upgrade since the ’90s

Tech in the tunnels: Muni train control system gets biggest upgrade since the ’90s

Muni’s computerized automatic train-control system is getting an upgrade.

And it’s about time, too, officials say. Muni’s mainframe harkens back to the mid-1970s — making its contemporaries Atari’s “Pong” and floppy disks.

“We’re taking advantage, if you will, of more and more technology,” said John Haley, director of transit at the San Francisco Municipal Transportation Agency.

The stakes are high, as 1998’s Muni Meltdown hasn’t been forgotten. That’s when a computer upgrade went horribly wrong and ground more than half of Muni’s light-rail fleet to a halt.

Muni riders were furious. The San Francisco Examiner’s headline after the delays read “Muni Hell on Wheels.”

For better or worse, the Muni Meltdown made the SFMTA skittish to upgrade its computerized train-control system all at once, Haley said. Instead, it has done it piece by piece, and now the agency is at a turning point in making some of those upgrades.

Muni is currently in the middle of overhauling its train-control systems. Some upgrades are complete, while others are set to finish by November.

The first of the upgrades required rewriting the software that controls trains on the railway, allowing trains to automatically berth next to one another, when two trains appear at a station simultaneously.

The newest upgrades will mothball the myriad banks of clicking copper to enable the newer relay system, only 2 feet wide, to monitor the tracks on its own.

As Haley led The Examiner on a tour of the agency’s old mainframe, he said the transition will be much smoother than in 1998.

In a dank walkway tucked away inside the Van Ness Avenue Muni station is the mainframe that controls Muni trains underground.

Relay racks extend down a cramped hallway nearly 15 feet deep, where dozens of 4-inch copper filaments clatter up and down like teeth. The speedy “click, clack” sound signifies a smooth running Muni light rail automatic control system, which commands the J-Church, K-Ingleside,

L-Taraval, M-Ocean View, N-Judah and T-Third Street trains through the tunnels. The clapping beat signifies the proper alignment of the railway’s 83 axle (controls) and numerous other track switches.

“We have the chance about 2,000 times a day to screw up,” said Haley, referring to the number of entries light-rail vehicles make into tunnels every day.

Rail cars are under operator control outside the tunnels. To ensure proper transition to full computer control, drivers must guide their trains with precision.

If the train enters a tunnel at the wrong speed, Haley said, “The train slows down to under 10 miles an hour and it limps through the subway … everything behind it gets delayed.”

When this happens, or a track switch is damaged, a green relay cable running the length of the underground track is tripped. This signals the maintenance crew, which then checks each ticking copper relay for those with an off beat, or ones that have stopped altogether.

Many more systems must be checked: Above the clattering copper run hundreds of wires in and out of electric ports, resembling the back of a television multiplied several times.

The relays “tell us what they see,” third-year maintenance crew member Arvin Geronimo said.

Crews maintain and inspect this substation on many nights in any given week, as the older relays are also more susceptible to malfunction.

“Last year, we had water up to here” Haley said, gesturing with his hand to his waist. The relay system is under a city drain, he said. A yellow tarp is strung at the roof of the relay room, while sandbags, a bucket and more tarps are tucked away in the corners.

“Because you have all five lines going through the subway,” Haley said, “this has turned out to be the biggest area of vulnerability.”

Jim Kelly, deputy director of transit, said that just like trying to service an old MS-DOS computer, the components and experts for the aging relays faded away with time.

“These things are hard to replace,” Kelly said. “You can’t just go to Relays R Us.”

The old system runs simultaneously with the new. A smaller, 2-foot-wide version of the relay racks stands next to a metal cabinet with more recognizable modern computer components. Just around the corner, a simple black laptop is tucked into a cupboard, a control system for an essential upgrade.

“By going to the new system, reliability is much higher,” Kelly said. But this slow upgrade is just the start of the SFMTA’s biggest challenge. As hundreds of new light-rail vehicles will soon join the fleet, Muni’s main train-control computer system must be upgraded as well. v Kelly says they’re not sure yet what that new “brain” of Muni will look like.

The crew that maintains and inspects the relays every night said it is hopeful for the new system — but it also respects the old one.

“I trust it to run another 20 years,” said Hoa Huynh, a 20-year maintenance crew member. But, he said, “It’s slow. Very slow.”

Bay Area Newscomputer controlMuniSFMTATransittransportation

If you find our journalism valuable and relevant, please consider joining our Examiner membership program.
Find out more at

Just Posted

Passengers ride the 14-Mission Muni bus on Friday, March 12, 2021. (Jordi Molina/Special to S.F. Examiner)
Transit officials fear Free Muni pilot could hurt already-strained service levels

Supporters say fare cuts could increase ridership, help low-income residents

The vacant property at 730 Stanyan St. in the Haight currently houses a safe sleeping site for the homeless. <ins>(Kevin N. Hume/S.F. Examiner)</ins>
Affordable housing project on former McDonald’s site delayed as city adds units

The start of construction on an affordable housing project on the site… Continue reading

Visitors read a notice hanging on the Polk Street entrance to City Hall on Thursday, March 26, 2020, shortly after the building was closed. (Kevin N. Hume/S.F. Examiner)
City Hall reopening to the public on June 7 after long closure due to COVID-19

San Francisco will reopen City Hall to the public on June 7… Continue reading

Historic streetcars are undergoing testing on The Embarcadero to prepare for their return to service on May 15.<ins></ins>
What to expect for Muni Metro’s relaunch on May 15

Significant service restoration includes downtown subway tunnels and historic streetcars

San Francisco 49ers quarterback Jimmy Garoppolo (10) and running back Raheem Mostert (31) celebrate after Mostert scores his fourth rushing touchdown in the third quarter as the 49ers take on the Green Bay Packers in the NFC Championship game at Levi’s Stadium in Santa Clara, Calif. on Sunday, Jan. 19, 2020. (Kevin N. Hume/S.F. Examiner)
49ers on prime-time TV five times in 2021

Usually a team that finishes in last place and won only six… Continue reading

Most Read