For almost two weeks, Muni’s NextBus prediction system has left more than 700,000 daily riders stranded, left in the dark as to when their next vehicle is coming.
Many have publicly called the meltdown a failure of the San Francisco Municipal Transportation Agency.
In response, the SFMTA has claimed a transition of telecom giant AT&T’s networks occurred earlier than they anticipated, hampering plans to circumvent any NextBus woes.
But Muni’s NextBus system outages may have been avoidable with a radio system first proposed in 2008 — and long delayed — records reviewed by the San Francisco Examiner show.
And emails obtained by the Examiner show AT&T’s pushback on the narrative that they are to blame for the NextBus shutdown and reveal further disarray as the SFMTA scrambled to restore Muni predictions.
As early as Jan. 3, complaints of the technology’s failure began to stream into the transit agency.
“Next Bus said the T would be at 3rd and Oakdale in 18 minutes,” wrote one customer, Richard DeWilde. “Then it pushed back to 21. Then to 39. When it was at 21 again, it jumped back to 39 again.”
Mayor Ed Lee also expressed dismay.
“The mayor is as frustrated as Muni riders that this occurred and has yet to be completely fixed,” the Mayor’s Office told the Examiner. “He has asked SFMTA, the independent agency that oversees Muni, for daily updates in this issue.”
At first, the agency was in the dark as to the cause of the outage, and emails circulated within the agency with the subject heading “Possible nextbus issue” concerning the T-Third train line.
By Jan. 4, the SFMTA and NextBus, which is owned by Cubic, had identified the problem — AT&T began shutting down service on its 2nd Generation cellphone towers, known as 2G, to move its network to rely primarily on 3G service.
SFMTA repeatedly asserted this publicly.
But in emails to the SFMTA, Theodora Vriheas, a lobbyist for AT&T, quietly disagreed that the telecom giant had not warned the agency beforehand and said they had “gone out of our way” to provide the SFMTA and NextBus with regular updates.
In transit circles and private transportation message boards, experts wrote that the 2G transition was known as early as 2012, which SFMTA spokesperson Paul Rose confirmed to the Examiner.
Lisa Walton, SFMTA’s tech chief, wrote to Vriheas that AT&T told them they would “resume work” on 2G transition at the end of January, instead of the beginning — leading to the NextBus outages.
While that may be true, public records show the SFMTA had proposed technology which may have prevented the meltdown for close to a decade.
Upgrades long delayed
The SFMTA had a proposal on the table eight years ago to create an internal network for Muni that would allow the fleet to not be dependent on the AT&T network service for bus arrival times.
That radio network is called the CAD/AVL, for Computer Aided Dispatch and Automatic Vehicle Location system.
That system was set for completion in late 2015, SFMTA records show, though the Examiner was unable to verify if that was the first proposed completion date.
Though SFMTA is currently pursuing such a system from Xerox/ACS Orbstar, the SFMTA had a proposal in 2008 to obtain such a system from Harris Corporation, which received further approvals from the SFMTA Board of Directors in 2012.
SFMTA has now switched to network company Xerox for the CAD/AVL systems, even as transportation agencies nationwide are pursuing different companies.
The Santa Clara Valley Transit Authority, for instance, had a Xerox-provided CAD/AVL system in place since 2001, but did not partner with Xerox for their newest CAD/AVL system. Gary Miskell, chief information and technology officer at VTA, told the Examiner that Xerox did not even submit a new bid to VTA — perhaps odd, considering they partnered on a radio system for so long.
“VTA was looking for a State of the Art CAD/AVL system” and opted to pursue a contract with Clever Devices, Miskell said.
Following the outage, Muni mechanics and NextBus employees raced through many nights to install modems on as many as 500 buses and many more light rail vehicles to get them back on the NextBus system.
Despite their efforts, the teams encountered unexplained slowdowns.
“I was told that they got off to a slow start as the first [six] or so units they installed would not power up. [Nineteen] units is far less than I expected to be done by this AM,” wrote Scott Middleton, manager of rail maintenance at SFMTA, to his bosses Jan. 7.
By contrast, 54 modems were installed on Muni light rail vehicles the night of Jan. 9, emails show.
On Jan. 7, emails between NextBus and the SFMTA show as many as 150 wrong modems may even have been delivered to the agency.
The emails also reveal a rush to find any local supplier of modems, called VMXs, with the next “firm supplier” of them being as far as four weeks away, Blair Brown, operations director at Cubic, the parent company of NextBus, wrote to the SFMTA on Jan. 11.
That may mean a full fix for NextBus may be due in mid-February.