Transportation Authority Kept Secret Cyber Attack That Cost $600,000

Computer code error message

Photo courtesy Lawrence Monk

Print More

The Orange County Transportation Authority was struck with a major cyber attack in February that cost over $600,000 and disabled dozens of computer servers for days, including a total shutdown of email, voicemail and numerous other services.

The “ransomware” attack started around 1:15 p.m. on Thursday, Feb. 4, with malicious software taking control of 88 servers at the agency, according to spokesman Joel Zlotnik.

Those servers – which run email, voicemail, internal intranet, bus driver assignments, payroll, and about a dozen other applications – were held hostage by the cyber attackers, who demanded about $8,500 in ransom, Zlotnik said.

It took two and a half days – until around 11 p.m. on Saturday, Feb. 6 – for the servers to be restored.

“It was a significant disruption. Everyone in this [headquarters] building and everyone throughout [the transportation authority] relies heavily on email, on voicemail, and all of these other systems,” Zlotnik said. “There were a number of [IT workers] who didn’t even go home to get a couple of hours’ sleep.”

Transportation services were able to still function normally, Zlotnik said, and no personal information, such as credit card or social security numbers, was stolen.

The revelation comes amid growing attention to cyber attacks in recent years against companies and government agencies. The issue was front-and-center in the presidential election last week when it was revealed that the Democratic National Committee had been hacked, presumably by the Russian government, and 19,000 internal emails were released.

All in all, the Transportation Authority estimates that the attack cost about $660,000, including about $330,000 in labor costs for the agency and its contractors, as well as $218,000 in emergency contracts with Microsoft and San Clemente-based CISO Share to fully clean out the malicious code, analyze the attack, and prevent more cyber attacks.

Brown Act Violation?

However, while Transportation Authority board members were notified of the attack in its immediate aftermath, the only public reference by officials was a vague announcement that the agency had experienced “technical problems” and “technical issues.”

At no point in the six months since it happened, even after the vulnerability was fixed in early March, has the agency issued a specific announcement regarding the attack or put it on a public meeting agenda. The board approved the $218,000 in emergency contracts with Microsoft and CISO during a Feb. 22 closed-session meeting.

This lack of transparency in one case amounted to an apparent violation of the state’s open meetings law, known as the Ralph M. Brown Act, said Terry Francke, general counsel for Californians Aware, who is one of California’s foremost experts and advocates on open government issues.

Francke said the Transportation Authority board’s closed-door purchase of $218,000 in services in response to the attack was unlawful because it “was not on the agenda and it was authorized in an unlawful closed session.”

The Transportation Authority disputes that, saying they were “in full compliance with the Brown Act and we completely disagree with Mr. Francke’s opinion.”

In interviews over the past week, Zlotnik explained the agency’s decisions to spend the $600,000 to revamp the system rather than pay the $8,500 ransom, and to not let the public know there had been an attack.

“The FBI opposes paying ransom for cyber attacks, and so does [the Transportation Authority],” he said. “If we pay ransom to a criminal, there is no guarantee that our servers would be released,” and the agency would likely be a target again because the attackers know they pay up.

The closed-discussion and approvals were done in a way that didn’t give any clues that an attack had taken place. Zlotnik said the agency didn’t announce it because doing so might invite further attacks, and cited the open meeting law’s exemption for security threats as justification for the closed session discussion and action.

“The last thing we want to do is make a public announcement…Why would you let people know that your systems are compromised? It would invite, potentially, other people to hit you,” he said. “I think we did everything that we should have done.”

However, this position appears to be at odds with previous statements by Transportation Authority CEO Darrell Johnson about the importance of being upfront with the public about cyber attacks.

When he was the agency’s deputy CEO, a transportation publication paraphrased him as saying that “if an organization’s electronic security is breached and information is lost or stolen, or if service is disrupted, the organization is at risk of losing the trust of its customers, constituents and the general public.”

“To safeguard that public trust, [the Transportation Authority] maintains a disaster management and recovery plan in the event that security is breached. The plan includes steps to notify the public of what happened and how the agency will rectify the situation,” Johnson said, according to the article in Progressive Railroading.

“We really want to make sure we have a professional and positive image to present to our constituents and the taxpayers, and that we ensure public trust,” Johnson added.

Zlotnik said this situation was different from the one Johnson was describing, in that services to the public weren’t disrupted and data wasn’t stolen.

“What Darrell said was true and it remains true today. Again, in this crime against OCTA, information wasn’t lost or stolen and service wasn’t disrupted. If that had been the case, those impacted would have been notified,” Zlotnik said, adding that he would have explained the February attack sooner if anyone had asked about it.

Zlotnik also suggested that Voice of OC ask the FBI and the county’s intelligence assessment center about what they recommend on whether to notify the public about attacks.

FBI spokeswoman Laura Eimiller said her agency doesn’t have general advice about whether government agencies should publicly disclose cyber attacks, and that such decisions are up to the organization that is attacked.

And the Orange County Intelligence Assessment Center “does not provide advice to public agencies on disclosing cyber attacks,” according to Lt. Mark Stichter, a spokesman for the county sheriff’s department, which is the lead agency at the center.

The Transportation Authority attack was referred to federal authorities for investigation, Stichter added.

Transportation Authority Chairwoman Lori Donchak, who’s also a San Clemente councilwoman, didn’t return a phone message asking if she agreed with the decision to not tell the public about the attack.

Francke, the open government advocate, said the security exemption used for the closed session only allows discussions with certain law enforcement officials, agency lawyers, “or a security consultant or a security operations manager.”

The closed session was held between the board and the Transportation Authority’s top technology official, Chief Information Officer William Mao.

“A conference with an information officer would not justify a closed session,” Francke said.

Francke also took issue with the approval of $218,000 in contracts during the closed session, which weren’t listed on the meeting’s agenda. The exemption used does not allow for such approvals, he said.

Another open government advocate, Kelly Aviles, agreed.

“It was unlawful to approve the purchase orders under that closed session exemption,” she said. “The remedy at this point would be to submit a cease and desist demand to prevent them from using that closed session for similar circumstances in the future.”

In a statement, the Transportation Authority said such claims are wrong and that they fully complied with the law.

“Closed sessions are allowed under the Brown Act for exactly this type of situation. It would be irresponsible, if not negligent, to publicly expose our security weaknesses and vulnerabilities that were exploited by the hackers,” said the statement.

“Our chief information officer manages our cyber security operations and discussing this in closed session with him is entirely appropriate and permissible under the Brown Act. [The Transportation Authority] properly listed the closed session item on the agenda,” it continued.

“We agree that public access to information should only be limited in very narrow cases, and this is very much one of those cases.”

The Transportation Authority is footing the full $660,000 bill for now. But Zlotnik said staff believe it’s likely the agency will be fully reimbursed, and they’re “pursuing every avenue to ensure that it happens.” The agency has cyber-security insurance for this kind of attack, Zlotnik said.

Correction: A previous version of this story incorrectly identified one of the technology companies hired to respond to the attack. The correct company is CISO Share. We regret the error.

Nick Gerda covers county government and Santa Ana for Voice of OC. You can contact him at ngerda@voiceofoc.org.

  • Pingback: Hacker Attacks, Data Breaches and Claims | Cyber Insurance News()

  • octaxpayer

    I thinking about the cost and they got ripped off to fix 88 servers at $600k is over charge. They could hire a security guy for six years to help stop these attacks and the know how to fix just in case they do get hit. I wonder if they had backups as this is a
    bout the only way to recover from this type of attack.

  • octaxpayer

    The county and all the agencies within OC tends to hide all they can that makes them look bad including the security vulnerabilities. Security assessment in the past at the County have found multiple vulnerabilities and IT staff were told to keep this hidden and basically they did nothing to correct the situation. The County is a large target to which they have never even addressed policies for security. They have and will continue to have breaches that go unreported. Some of the information is highly sensitive and by disclosure law needs to be reported..

  • Jacki Livingston

    Puh-leeze. This happens all the time. SSA had similar attacks and then accused the employees of putting magnets on their computers. It is the blind leading the naked down there,

  • OCservant_Leader

    How do we not know this wasn’t an inside job triggering the emergency 600K flow towards secret contacts?

    A 8K threat triggered a 600K expense?

  • David Zenger

    Speaking of the OCTA, maybe somebody can explain why Metrolink trains are now being pulled by BNSF engines.

    How much does that cost?

    • It apparently has to do with a design flaw with Metrolink cars that might have caused a fatal derailment last year. The cost for Metrolink to lease the BNSF freight engines is $18 million per year, according to the LA Times: http://www.latimes.com/local/california/la-me-adv-metrolink-engines-20160501-story.html

      – Nick Gerda

      • David Zenger

        Yes, I know. And who’s paying for that? Not the people who bought the faulty engines, we may be sure of that.

        The transportation agencies are a black hole of waste and unaccountability.

    • calwatch

      You mean “the Transportation Authority”, which apparently is a unique Voice of OC style.