A shadowy remote hacker using technical cunning to poison a US city sounds like the stuff of a Hollywood thriller. But in reality, security experts have been warning about such an attack for years. So when the worst case scenario nearly played out in Oldsmar, Florida earlier this month, the cybersecurity community was perhaps less surprised than the public and mainstream media.
That shouldn’t detract from the gravity of the incident. But as we learn more about what actually happened, and what could happen in critical national infrastructure (CNI) facilities across America, it becomes clear that there are some key lessons which should be learned. SCADA security is challenging, but best practices will go a long way to mitigating risk.
What Happened in Oldsmar?
The first most people heard of the attempted mass poisoning was a hastily arranged press conference with officials from the city of 15,000, located a few miles from Tampa. They claimed that an unnamed attacker had somehow managed to hijack a computer at a local water treatment plant. They used this access to alter the amount of sodium hydroxide added to the water by around 100-fold, to dangerously high levels. Fortunately, the IT worker sat watching this unfold, and was able to immediately return the levels to normal once the attacker had logged off.
In any case, it would have taken more than a day for the sodium hydroxide to enter the water supply, and redundancies in the system would have spotted the change in pH level and sounded the alarm, according to Oldsmar mayor, Eric Siedel.
"...these bad actors are out there."
“The important thing is to put everybody on notice,” he warned at the press conference. “That’s really the purpose of today, to make sure that everyone realizes that these bad actors are out there. It’s happening, so take a hard look at what you have in place.”
Security Fail
In fact, from a cybersecurity perspective it should have been the managers of the treatment plant that looked closely at what they had in place. From subsequent notices from the FBI and the Massachusetts government, we know of multiple security failings at the facility. These included:
- Use of the legacy Windows 7 operating system, for which patches are no longer available
- Use of remote access system TeamViewer to connect to critical infrastructure
- Use of the same password for all computers
- Computers directly connected to the internet with no firewall installed
The attack vector appears to have been TeamViewer, which allowed the intruder to remotely hijack the computer and issue the commands to the plant’s SCADA system. The FBI reportedly issued an alert warning about the use of such tools in CNI facilities.
"Beyond its legitimate uses, TeamViewer allows cyber actors to exercise remote control over computer systems and drop files onto victim computers, making it functionally similar to Remote Access Trojans (RATs)," it said. "TeamViewer's legitimate use, however, makes anomalous activity less suspicious to end users and system administrators compared to typical RATs.”
It remains to be seen if the individual simply guessed or cracked the password in question, or if it was a disgruntled insider at work, as some have speculated.
Critical Problems
In fact, there is a bigger picture here. SCADA systems are present in most CNI facilities as a way to control industrial processes. But ever since the Stuxnet campaign over a decade ago showed the world the damage that could be done via remote attacks on such systems, experts have been warning about their security deficiencies.
The bottom line is such systems were designed before an age of global cybercrime and audacious state-sponsored attacks. Typically they were air-gapped from the internet and relied on “security-by-obscurity” to keep them safe. As such systems were upgraded with connectivity, remote attacks became a real threat. Yet vendors are slow to release patches, and in any case systems are difficult to upgrade – often because they are mission-critical and can’t be easily taken offline to test updates. Many of the legacy communications protocols used by these systems were not designed with security in mind, and proper IT-OT segmentation is often lacking.
Legacy communications protocols used by these systems were not designed with security in mind.
This opens the door to cybercrime (especially ransomware), cyber-terrorism and, most commonly, state-sponsored attacks. Russia seems to excel in this area and has been probing US CNI for years, including its energy grid, water system, and even nuclear facilities. The worry is that such attacks are “pre-positioning” malware to be activated at a later date, such as during a geopolitical crisis. The Kremlin has already shown many times over the destructive potential of attacks: notably two sophisticated raids in December 2015 and 2016 which led to mass power outages in Ukraine.
A Ticking Time Bomb
In a 2020 report, the Cyberspace Solarium Commission warned that America’s water infrastructure is dangerously fragmented. Although decentralization adds a measure of resilience, the commission claimed that this also makes it harder for government to roll out and enforce standards and best practices to all 70,000 separate water utilities.
“Gaps in utilities’ network configurations, insecure remote access systems, and outdated training regimes are just a few of the vectors through which Americans’ water infrastructure is vulnerable to cyber-enabled exploitation,” it noted. “Malign actors have already attempted to breach water infrastructure systems, and they could eventually exploit these vulnerabilities to disrupt or contaminate the American water supply.”
Now that these fears have been partially realized, it is hoped the federal government will put more money and effort into enhancing standards across the board to improve baseline security. Europe is way ahead of the US here in that its NIS Directive mandates best practices for organizations in various CNI sectors, with regulators able to wield huge fines of up to 4% of global annual turnover for non-compliance.
What You Can Do Today
However, the bottom line is that CNI operators, especially in the water sector, cannot wait for government action. Although cash is often lacking, especially since the pandemic, there are many simple steps they can take today to reduce cyber risk. These include:
- Multi-factor authentication for all accounts, including Remote Desktop Protocol (RDP) and other remote access systems
- Enforce a policy of “least privilege” to reduce cyber risk further
- Anti-malware and firewalls must be up-to-date and properly configured
- Network audits to isolate or apply virtual patches to any systems that can’t be updated
- Audits to understand where remote access systems are being used and whether they are necessary
- Enhanced staff training to spot and report attacks/social engineering
- User behavior monitoring or similar to spot insider or external threats
- Effective, risk-based patch management programs
- Proper onboarding and offboarding of staff so access rights are quickly revoked once staff leave
- Network segregation to contain the fallout of a breach
- Provide dedicated, secure laptops to remote workers rather than allow personal device use
The more sophisticated threats will require a more sophisticated response. However, hopefully this near miss is the wake-up call the US authorities need to enhance security best practice in CNI.