City of Lodi Switches to Rubrik to Bolster Its Disaster Recovery Following Ransomware Attack with Previous Backup Solution

Challenges

Previous ransomware attack disrupted critical municipal services

In 2018 and 2019, a series of ransomware attacks hit the city of Lodi. Hackers used malicious software to target Lodi’s phones and financial services, crippling the city’s ability to access swaths of its data. In 2019, extortionists demanded 75 bitcoins, approximately $400,000 at the time of the attack. The city followed guidance from a specialized cybersecurity team, and following the attack, a team of security and legal experts conducted a series of forensic audits. No public information was compromised as a result of the ransomware attack.

“A few years back, we were hit by three attacks in three months by the same ransomware. The attack significantly impacted our municipal services. Our objective was to prioritize the recovery process to get critical services back up as soon as possible. However, after the second attack, our CAD dispatch service went down. It took our CAD vendor at the time four days to get us 100% back up and running,” said Buecher.

4 weeks to recover and large data loss due to prior backup solution

“The recovery process with our previous backup solution was extremely slow and tedious. We first had to physically run to the data center and unplug our backup unit. It took us weeks to isolate and mitigate the infection. From the time we were hit to the time we were 95% recovered, it was about a month to completely restore over 100 virtual machines. On top of that, the entire recovery process was extremely manual. Due to the way our previous backup solution was implemented, we were forced to restore one virtual machine at a time and required all hands on deck,” said Buecher.

Another challenge was the complexity of restores. “With our previous solution, our only option was to perform file-level backups. As a result, during the attack, we had to rebuild the data on the machines from scratch instead of simply restoring the machines. That process of rebuilding all the virtual machines was the most time-consuming aspect,” said. Matthew Casson, Network Administrator at the City of Lodi.

Lastly, their previous approach failed to notify them of backup failures, resulting in one week of data loss for their ERP database. “The second ransomware attack took down our entire ERP. When we came in that morning, we realized the ERP wasn’t responding, and we couldn’t get into it. When we went to reboot that instance, the whole instance became encrypted. As a result, we didn’t get a chance to pull the data off that database,” said Buecher. “Our ERP vendor at the time was supposed to take nightly backups for disaster recovery. However, when we reached out to them following the ransomware attack, we learned our backup system had failed, and the latest copy was seven days old. It was frustrating that no one had told us and to discover the unnecessary data loss.”

The Results