Russian hackers broke into a server at the city’s Administration for Children’s Services at least five times last year, according to documents obtained by WNYC under the state  Freedom of Information Law. The agency is responsible for oversight of more than 100,000 vulnerable children. It took six months after the first attack for anyone in the city to even notice.

The documents show the server contained ACS's "preventive services client information." City officials won't be more specific about the information on the server.

The details of the incident have never been publicly reported, and city officials say there is no evidence any valuable data was stolen.

“Really it wasn’t that big a deal," said Anne Roest, the commissioner of the Department of Information Technology and Telecommunications, known as DoITT. "We engaged the agency. We made sure we locked down the network and the servers so that if there was something there it couldn’t spread.” 

But the attacks exposed flaws in the city’s cybersecurity. Documents and interviews show that despite years of efforts to consolidate digital operations, the city still runs on a patchwork of computer systems and agency turf wars could be leaving the city vulnerable.

The stakes: Every month the city faces 80 million external threats such as phishing emails and denial-of-service attacks, according to DoITT’s own estimates.

“It’s worse than ever,” said Dan Srebnick, who spent 14 years at DoITT and was the city’s chief information security officer when he left government in May 2013. “The risk level has really gone from mischief to, I think, international terrorism and crime.”

Criminals know that state and local governments collect and store a lot of personal information, said Dan Klinedinst, a vulnerability researcher at Carnegie Mellon University's CERT Division, a cybersecurity research and consulting center. An agency like ACS could be an attractive target.

“There’s a sort of a trend of looking for identity information like social security numbers and stuff that belong to children. The guess is that’s because people are less likely to be monitoring their children for identity theft,” Klinedinst said.

The break-In

Investigators found the Russian hackers broke in to an old server at ACS, one of the agencies still operating outside of DoITT’s secure infrastructure. The agency is working on moving its system over, officials said.

Roest said the first attack appears to have been in February 2014. DoITT spotted the intrusions in August 2014. Roest and the ACS commissioner sent a joint email after 10 p.m. on Aug. 20 alerting First Deputy Mayor Tony Shorris to a breach on the server housing preventive services client information.

He wrote back shortly after.

“We need to understand as quickly as possible if this breach could have allowed access to any other NYC systems,” Shorris wrote. The city redacted the rest of the email before turning it over to WNYC.

Internal emails show forensic investigators spent the next few weeks trying to piece together what happened. DoITT brought in forensics firm Foundstone to assist. They also notified the NYPD and FBI.

The forensics investigation was slowed at times by the age of the system.

“(W)e are dealing with obsolete systems and it is hard to get hardware, software and skills needed,” Roest wrote in one email.

Ultimately, investigators determined that hackers accessed basic information.

"We have confirmed today that the actor had accessed and viewed all the keys that would provide the actor with the ability to pull all data from the fiscal DB. However, at this time, there are no indications that the keys were used for anything other than for the fiscal summary data," according to an internal email sent to Roest on Sept. 3. The identity of the sender is redacted.

There was “no evidence of successful acquisition” of personal information, according to the final report on the attacks.

Forensic experts use such careful wording because it can be difficult to definitely prove nothing was stolen in an intrusion. Good hackers might cover their tracks and it can be difficult to piece together what happened without, say, good logs that record what happens on a system.

DoITT’s Commissioner Roest sounded confident no harm was done.

“I think people are hedging their bets when it comes to how firmly we’ll say this, but I’ll say it pretty firmly: there was no breach,” Roest said. “I mean we had experts from the highest-rated commercial cyber firms in here, we had our federal partners in here, and nobody is ever a thousand percent sure that there wasn’t a breach on any system.”

DoITT wouldn’t provide any technical details to show why Roest is so confident. A spokesman said releasing such information is a security threat.

The Administration for Children’s Services did not notify child services providers about the hacking incidents. Officials at the agency declined an interview request, but did provide a statement.

“At the completion of DOITT’s work to address the incident, ACS received assurance that there was no evidence of data exfiltration," the statement said. "Pursuant to DOITT’s advice, and guidance from our state oversight agency, ACS proceeded to notify our providers that the system was taken off line for maintenance.”

Several service providers said they first learned of the break-in from WNYC.

“I am deeply troubled," said Karen Freedman, executive director of Lawyers for Children, an organization that represents foster kids in court. "I’m actually stunned that I haven’t heard about it, that there hasn’t been any publicity about this at all.

“Any time there’s any chance that a young person in foster care who is already vulnerable, may be vulnerable to an additional form of attack…those responsible for caring for them should absolutely be notified so that they can be on alert and protect the rights and interests of those young people.”

Weak spots

It’s unclear who is in charge of cybersecurity citywide.

DoITT says they’re in charge. A 2006 Memorandum of Understanding transferred cybersecurity oversight to the agency from the city's Department of Investigation. The MoU gives DoITT the responsibility to develop citywide security procedures and to work with agencies to review their systems.

But records and interviews show that some agencies are slow to take DoITT’s security recommendations — and others simply ignore the agency.

For example, a 2010 audit by the city comptroller found some agencies launch web applications without bothering to go through DoITT’s Security Accreditation Process, designed to spot weaknesses a hacker might exploit.

“When I was in the role of managing security for the city, there were agencies that were a pleasure to work with and there were agencies that told me in no uncertain terms where to go,” said Srebnick, DoITT’s former top cybersecurity official.

It’s unclear if that’s changed. The comptroller’s office has not followed up on its 2010 report. It was the last audit the office has done on cybersecurity.

Last summer, DoITT contacted the Department of Investigation to discuss security issues, according to a document obtained through a separate records request.

One concern was that five large agencies weren’t using security products DoITT licensed from a major software company. The software would allow DoITT to see security vulnerabilities in those agencies.

The Department of Investigation found that some agencies were slow to install the programs — and two said they didn’t want DoITT’s oversight and would handle their own cybersecurity.

Commissioner Roest said the city has never experienced a major data theft and as long as the agencies have their own protection, it’s OK.

“I’m confident that those agencies are safe,” she said.

The Department of Investigation, however, asked WNYC not to name the agencies for security reasons.

Srebnick said disputes between DoITT and agencies are a problem, and that elected leaders need to do more.

“I think that ultimately City Hall needs to take a lead in telling the commissioners of the agencies that information risk management is important, that the mayor views it as important, that his staff will follow up on the importance," he said. "That may begin to get the message out.”

The mayor’s office declined an interview request, as did James Vacca, chair of the City Council’s Technology Committee. His spokesman said Vacca is still getting up to speed on the issue. The committee last held a hearing on cybersecurity three years ago.

 

ACS Cyber Intrusion - CAT 1, Forensic Findings Report Redacted