Software update takes out nuclear power plant

On March 7 the Hatch nuclear power plant near Baxley, Georgia was forced into an emergency shutdown for 48 hours after a software update was installed on a single computer. The trouble started after an engineer from Southern Company, which manages the technology operations for the plant, installed a software update on a computer operating on the plant's business network.

The Washington Post has a very good article by Brian Krebs, which is worth reading in its entirety. Highlights:

According to a report filed with the Nuclear Regulatory Commission, when the updated computer rebooted, it reset the data on the control system, causing safety systems to errantly interpret the lack of data as a drop in water reservoirs that cool the plant's radioactive nuclear fuel rods. As a result, automated safety systems at the plant triggered a shutdown...

Company technicians were aware that there was full two-way communication between certain computers on the plant's corporate and control networks. But she said the engineer who installed the update was not aware that that the software was designed to synchronize data between machines on both networks, or that a reboot in the business system computer would force a similar reset in the control system machine.

"We were investigating cyber vulnerabilities and discovered that the systems were communicating, we just had not implemented corrective action prior to the automatic [shutdown]," Phillips said. She said plant engineers have since physically removed all network connections between the affected servers.

Computer security experts say the Hatch plant incident is the latest reminder of problems that can occur when corporate computer systems at the nation's most critical networks are connected to sensitive control systems that were never designed with security in mind.

Specifically, experts worry that vulnerabilities were introduced into the systems that regulate the electrical grid as power companies transferred control of generation and distribution equipment from internal networks to supervisory control and data acquisition, or SCADA, systems that can be accessed through the Internet or by phone lines, according to consultants and government reports.

The move to SCADA systems boosts efficiency at utilities because it allows workers to operate equipment remotely. But experts say it also exposes these once-closed systems to cyber attacks...

Joe Weiss, managing partner at Cupertino, Calif.-based Applied Control Solutions, said Hatch is not the only plant that has suffered this type of unusual event. But he said it is one of a handful of public events of this type because the Nuclear Regulatory Commission documents all unusual events, in contrast to non-nuclear facilities that do not make their unusual events public.

"Consequently, it is expected that non-nuclear facilities have experienced similar events," Weiss said. "The Hatch event illustrates the unintended consequences that could occur when business information technology systems interconnect with industrial control systems without adequate design considerations." ...

Weiss said many people in charge of SCADA systems have sought to downplay the threat that hackers pose to these complex networks. But he cautioned that internal, accidental cyber incidents at control system networks can be just as deadly as a carefully planned attack from the outside...

"To people in the IT world, cyber means 'attacks,' but what I tell people is that in our world the predominant cyber events are unintentional," he said. "The flip side of that is if it can happen unintentionally, it can probably be caused intentionally and be a whole lot worse."

News of the Hatch incident also comes as the cyber-security posture of the electric and nuclear power industry is coming under increasing scrutiny from Congress and government investigators. Last month, the Government Accountability Office issued a scathing report about cyber security weaknesses at the Tennessee Valley Authority, the nation's largest public power company and operator of three nuclear plants, including Browns Ferry.

This reinforces the points made in my previous post on protecting the BPS.