Orange County officials failed to implement essential safeguards to protect county computer systems, which left the county unnecessarily vulnerable to hacking and other malicious activity until the problems were uncovered in recent months by a comprehensive audit.
“We found that physical and [software access] security to data and programs WAS NOT appropriate, approved, managed, maintained, and adequately supported,” the auditors wrote in their draft report, which was obtained by Voice of OC.
The audit looked at county systems in calendar year 2016 but some of the problems apparently existed for years. There had been no audit as comprehensive as the current one for more than a decade, according to the draft report.
The county’s central information technology (IT) officials fixed many of the issues after the auditors discovered them, according to what they told the auditors. But some key vulnerabilities apparently haven’t been fixed yet, and the audit raises larger questions about the county’s cybersecurity leadership, policies and practices.
Among the wide-range of problems auditors discovered, which officials said were later corrected: several critical computer systems did not have up-to-date antivirus software, despite officials being able to automatically deploy security updates across the network. And former workers continued to be allowed access to the back-end of county computer networks even after they left their jobs.
In one case that was fixed after auditors discovered it, the county failed to revoke high-level network access from a former county contractor who had been terminated the prior year. By logging in, he would have been able to enter the county network undetected, weaken protections against cyber attacks, and cover his tracks, they wrote.
“Of further concern,” the auditors wrote, the former contractor tried to test the security of county systems through what is called a “penetration test.” The draft audit report didn’t say whether he tried the “penetration test” before or after he was terminated.
And almost 25 percent of security access badges to enter the county’s sensitive central data center were assigned to employees who either were inactive or terminated. This too was fixed after the auditors found it, according to what county officials told the auditors.
In the big picture, county officials haven’t implemented a comprehensive top-to-bottom strategy for preventing cyberattacks known as a “cybersecurity framework,” which is considered a nationwide best practice, auditors found.
By not having this in place, they wrote, the county is less prepared to deal with “system breaches, virus attacks, ransomware attacks, unauthorized access to data,” and loss of information.
Auditors also found significant weaknesses in how the county oversees changes to the work information technology contractors do, which they said “could result in budget overruns,” missed deadlines, and problems with the reliability and performance of systems.
Those weaknesses include missing information about who approved contract work changes and when, which auditors said could result in unapproved changes being performed.
County officials declined to answer any questions about the audit’s findings, saying they won’t be commenting until a final report is issued.
“Once a final report has been provided, the County’s Information Technology department will review its findings and provide a response outlining security measures that have been put in place, while being mindful to not compromise the security of the County,” said county spokeswoman Carrie Braun.
“Until that time, it would be premature to respond to a draft of a report that could change before becoming final. The County’s Information Technology team will continue to provide innovative, reliable, and secure technology solutions that support County agencies and departments in the delivery of quality public services.”
Auditor-Controller Eric Woolery also declined to comment, citing the ongoing audit, but said his office will be able to discuss its findings after the audit is finalized.
Hackers have caused significant disruption to government agencies in recent years.
The Orange County Transportation Authority fell victim to a ransomware attack last year, when its computers were held hostage by malicious code. The agency didn’t pay the ransom but the OCTA’s insurance company paid more than $600,000 for consultants and staff to regain control of the computers.
And a global cyberattack in June forced several hospitals to shut down in the United Kingdom, when hackers seized control of their computers and demanded ransom payments.
The county audit, which was circulated among key county officials, looked at computer networks run by the county’s central IT office. Those systems are used by most county departments, like health and social services, probation, public works, and child support services.
Information on the computers includes data about residents using those services and other information entered by the county, including the office computers of the Board of Supervisors.
Among the serious problems found in the audit, which were corrected after auditors found them:
- Faulty password security for accessing a computer system that can weaken county firewalls and allow unauthorized access to the county’s network, including sensitive files and the installation of malicious code, known as malware. After the auditors found these problems, the password requirements were strengthened.
- Use of more than 15-year-old server operating systems that their manufacturer no longer provides security updates for. This increases the risk that already-known vulnerabilities with those operating systems could be used to install malware, like ransomware or spyware, or access sensitive data, auditors wrote. After the auditors’ review, the county’s central IT office told them it was starting to upgrade the operating systems.
- Several “generic” user accounts that were granted “complete control” of county computer networks, which removes the ability to hold individuals accountable for activity, and violates IT best practices and county policy, according to auditors. County officials were provided the opportunity to disable these specific accounts and only allow accounts that are specific to a single individual.
- An excessive number of security access badges issued (nearly 400) that allow people to enter a sensitive data center. Additionally, almost a quarter of reviewed access badges “were issued to inactive or terminated employees.” After the auditors discovered this, county officials said they revoked access rights for hundreds of accounts the auditors expressed concerns about.
- Failure to install security updates on Microsoft Windows computers. Additionally, county contractor Science Applications International Corporation (SAIC) was using an outdated server operating system that’s nearly 15 years old and “has known security vulnerabilities that cannot be rectified.” After the auditors discovered this, central IT officials corrected it, according to the draft audit.
Before this audit, the county hadn’t done a broad audit of its IT security in at least ten years, according to the report. For almost all of that time, the county’s auditors reported directly to the Board of Supervisors.
Then, in August 2015, the internal audit division was transferred to Woolery, who is an independently elected official. It was after Woolery took over that the auditors conducted their review of computer security and found widespread problems.
During the time county supervisors oversaw auditing, they were receiving significant campaign contributions from the same vendors that the recent audit found had oversight problems. One of them, Xerox Corp., funneled more than $12,000 in campaign contributions to supervisors before the vote.
Just over a year after Xerox was awarded the contract, county officials said the firm caused months-long installation delays and cost overruns expected to exceed $13 million. Xerox later sold its IT business to the French company Atos, which has kept the county contract.
The person in charge of county IT has changed several times in recent years. Christina Koslosky became chief information officer in March 2014 and left just over two years later, in June 2016. The county then had an interim director until this April, when Joel Golub was appointed to the job permanently.
In their review, auditors also found problems with the county’s decentralized approach to IT.
“Information Technology at the County has operated in a decentralized model since 1996, which has resulted in inefficient IT business process mechanisms that potentially impact reliability and services/support,” the auditors wrote.
The county is now moving to a more centralized model, known as “shared services,” that officials expect will provide better security standards and training, while also reducing costs.
On the positive side, the auditors found a series of strengths, including filling the permanent IT director position and the recent creation of a Cyber Security Team that has been put in charge of a new assessment of the county’s ability to withstand cyber attacks.
As for implementing the audit’s recommendations, the auditors plan to review that in a follow-up audit that would start six months after the final version of the audit report is released.
The final audit isn’t expected to be released for several months.
After the final audit is released, auditors wrote, the county’s Audit Oversight Committee and Board of Supervisors “expect that audit recommendations will typically be implemented within six months and often sooner for significant and higher risk issues.”
Correction: An earlier version of this story didn’t make it clear that OCTA didn’t pay ransom. Voice of OC regrets the error.
Nick Gerda covers county government and Santa Ana for Voice of OC. You can contact him at ngerda@voiceofoc.org.