The Friday after Thanksgiving, Muni’s top tech guru reclined in her pajamas in her Nob Hill home, playing a game of “Bubble Pop” on her iPad.
That’s when Lisa Walton, chief technology officer for the San Francisco Municipal Transportation Agency, received the email — Muni had been hacked.
“I was immediately drawn to the fact that it looked like a virus,” Walton told the San Francisco Examiner on Tuesday, in the first official recounting of the behind-the-scenes battle to fight back against the malicious software attack that struck on Thanksgiving weekend.
“Bubble Pop” would have to wait.
Walton headed to SFMTA headquarters to assess the problem.
“At that point, we went into ‘identify and contain’ mode,” she said.
By that time, a software pirate had splashed a message — “You Hacked, ALL Data Encrypted. Contact For Key(firstname.lastname@example.org)ID:681 ,Enter.” — across many SFMTA computers, some of which were visible at train stations throughout The City.
According to the alleged “hacker,” an employee at the SFMTA had mistakenly downloaded malicious software, known as ransomware, designed to scoop up important data and hold it under a digital lock and key for money. The “hacker” demanded 100 Bitcoin, a digital currency, which translates to about $73,000.
Walton contacted SFMTA management, and the agency contacted the FBI and the Department of Homeland Security.
One of the first priorities was restoring computers at Muni division yards, where buses are deployed. For the weekend following the attack, Muni operators told the Examiner they were assigned buses by handwritten paper — far afield from their usual computer printouts.
Muni service wasn’t affected, though the agency opened fare gates and shut down its train fare systems, so they were isolated from the potential spread of the malware.
All told, the SFMTA estimates it lost $50,000 in revenue from two days of free train rides during the attack.
Walton also assembled her teams specializing in networks, servers and desktops, which amounted to about 15 people.
“We went into disaster recovery mode,” she said. “Even if something wasn’t overtly affected, we needed to make sure, regardless.”
The team went to work fixing most of the SFMTA’s 8,000 computer systems remotely from their offices on Market Street. Using backup copies of the computer hard drives, they “re-imaged” most of the computers by the following Monday, Nov. 28.
A few stragglers — laptops taken home for the weekend or ones they missed at bus yards — took until the following Wednesday to fix. By then, the process was complete.
The SFMTA said previously it never intended to pay the ransom, but in Tuesday’s interview, Walton gave further illumination as to why.
“The hacker kept saying [they were ransoming] 32 gigabytes of data. That’s nothing,” Walton said. A gigabyte is a unit of measure concerning file sizes in computer storage. “We have 26 data stores at 300 [gigabytes] a piece,” she said, and “that’s just email.”
Still, Walton added, “We were being cautious, we didn’t take anything for granted.”