Thursday, May 25, 2017 At 11:07AM
It’s no secret that the devices that comprise process control systems are generally vulnerable to attack. This point has been made through endless research and has even been the subject of countless talks and trainings. Unfortunately, the personnel responsible for securing these networks often face significant challenges, most notably the difficulty in ensuring that devices and systems are configured securely and regularly patched without interrupting the process. In addition to this, security personnel often struggle in vain to sell the idea of security to the people in charge of the process who sometimes view security as more of an unnecessary burden, especially at lower layers of the process control network. In response to this, most of the focus has been placed on network segregation and establishing secure enclaves for sensitive process control systems. In this way, more effort can be placed on securing the barriers between the corporate network and SCADA / process control networks and enforcing tight access controls. In turn, many of the process control system assessments we’ve worked on have been almost entirely focused on determining the adequacy of network segregation efforts.
In spite of the importance of properly implemented network segregation and the interest in segregation testing, there isn’t too much information out there on performing them. The content of this blog post will detail some of the common gaps in network segmentation that we’ve seen during previous assessments. Hopefully, this information will help penetration testers who are looking to get into process control system assessments as well as give process security personnel an idea what attackers are looking for when trying to gain access to lower layers of the process control network from the corporate network.
As a general rule, network segregation assessments are performed from the perspective of an attacker that has already managed to compromise the corporate network in order to mimic real-world attack scenarios. Typically, this means either providing domain administrator access or performing the segregation assessment in conjunction with a penetration test of the corporate network.
The network segregation testing process typically follows the procedure outlined below:
- Information Gathering
- Entry point Identification
- Segregated Network Access
Information gathering
The process of information gathering centers around identifying any unprotected process control network-related information on the corporate network. This information can reveal intricate details about the process control network that can help an attacker gain access. Typically, this includes the following:
- Process-Related Personnel – Key people such as process engineers, operators, and process IT personnel are usually privy to a great deal of sensitive information related to the process control network. Searching Active Directory for employees that fall under the aforementioned roles can help to identify their workstation hostnames as well as any network shares they may have access to. This can help narrow down the information gathering process and reduce the overall amount of effort required to gain access to the process control network since focus can be placed on those hosts and shares.
- Network Diagrams and Design Documentation – This information can help an attacker understand the process control network at a fundamental level and provide hostnames and IP addresses that can serve as the basis for further information gathering. Moreover, network documentation may give an attacker a clear picture of the process control network, aiding them in deciding on which hosts to attack. After identifying corporate network file servers and information repositories like SharePoint, network documentation can typically found by searching for VSD, PDF, and DOC / DOCX files in directories or shares related to process IT.
- Process Control Network Documentation – This can include details on official remote access procedures, instructions on how to login to process-related hosts, and information on the technology in use (both operational and non-operational). In many cases, extremely sensitive information such as usernames and passwords are included in this documentation, making it invaluable on a segregation assessment. This documentation can also be found in process IT shares and directories as well as occasionally inside of process-related shares and directories.
- Network Device and System Backups – Firewall, router, and switch configuration files that are backed up on the corporate network can often serve as a road map for identifying a path to gain access to the process control network. Reviewing network device configurations can reveal which hosts on the corporate network are permitted access through remote access protocols such as RDP, VNC, or SSH. If backups of systems on the process control network can be found on the corporate network, they can be easily analyzed to extract password hashes and other sensitive data. This information can often be retrieved from IT management or network backup software as well as corporate and process IT network shares.
Entry point identification
One of the key pieces of information that should be obtained from the information gathering phase is the existence of any entry points into the process control network from the corporate network. Process control networks are rarely completely air gapped due to the fact that they may be located in remote or unfavorable working environments. As a result, remote access is a popular solution for managing devices and systems in the process control network. The following entry points can be found in most situations:
- Jump Box / Terminal Server – Access to the process control network from the corporate network is often permitted through a host that serves as a jump box. Usually a remote access protocol such as RDP, VNC, or SSH is used, although desktop virtualization software such as Citrix is occasionally employed instead. More security-conscious organizations tend to use remote access solutions that support multi-factor authentication.
Official procedures for remote access to the process control network can usually be found in documentation. Jump boxes or terminal services can also be identified from analyzing network diagrams or firewall configuration files for hosts that permit connections from the corporate network over typical remote access ports. Observation of process-related personnel’s corporate network workstations can oftentimes be an effective means of identifying the jump box and remote access protocol in use. - VPN Access – Select users, usually process-related personnel, can be granted VPN access directly into the process control network for remote management purposes. Similarly to remote access through jump boxes, multi-factor authentication may be employed to help thwart attacks against the process control network.
These users can sometimes be identified by an AD group created specifically for VPN users. Looking at the names and descriptions of groups where key process-related personnel are members is usually an effective means of identifying the process VPN group. Workstations of users in the process VPN group should be investigated for VPN interfaces or software as well as active connections into the process control network. These workstations can effectively be used as a bridge into the process control network. - Dual-Homed Hosts – In a typical process control network, several hosts on the corporate network may configured to be either dual homed or with extensive firewall rules that permit access into the process control network. Historian servers often fall under this category, due to the need to use process data for performance monitoring and improvement purposes on the corporate network. Gaining access to these hosts can often provide direct access into the process control network without having to use official remote access procedures. Even if historians are not provided extensive access into the process control network, observing their active network connections can reveal target IP address ranges.
The most reliable way to identify historians is to analyze network documentation and diagrams. Barring this, it’s not uncommon to find them by analyzing server hostnames (“HI”, “HIS”, or “HIST” are pretty widely-used naming conventions, for example). They may also be found through port scanning, although this can be a difficult task given the fact that there is little port standardization between vendors. However, many organizations with process control networks use OSISoft PI servers as historians, which can sometimes be identified by scanning for ports that are commonly associated with the software (https://techsupport.osisoft.com/troubleshooting/kb/2820osi8). In general, if the process control system vendor is known or discovered through reconnaissance, publicly-available vendor documentation should be researched for commonly-used ports to scan for on the corporate network.
Another effective method of discovering dual-homed hosts is sweeping across the corporate network using some form of remote command execution, such as WMI or PSExec, and extracting active network connections, network interfaces, and routing tables. This data can then be searched for any hosts that appear to have access to the process control network. However, this method is noisy, time-consuming, and can produce an excessive amount of information to parse through, especially on larger networks. Additionally, the IP address ranges used by the target process control network would have to be known in order to make this technique more feasible.
Segregated network access
With a list of potential entry points into the process control network, the next step is to investigate each entry point to determine how it might be possible to leverage common security issues in order to gain access to the process control network. The following issues are typically observed when conducting process control network segmentation tests:
- Insecure Password Practices – Passwords are a significant issue for personnel put in charge of managing the security of process control networks for multiple reasons. Most notably, it is difficult to encourage engineers and other users with access to the process control network to engage in secure password practices. Password reuse between the corporate network and process control network is extremely common since users are typically unwilling to manage multiple sets of credentials and feel that the risk of an attacker gaining access is small. After compromising the corporate network, the passwords for key process-related personnel should be cracked or obtained so they can be used for logon attempts on process control network entry points.Default or weak credentials are also very common. This is usually because these passwords are set by default as part of the vendor’s provided solution and never changed in order to avoid issues with deviating from the vendor’s default configuration. Oftentimes usernames and passwords will be the same (operator : operator, manager : manager, supervisor : supervisor are common examples) or some variation of the vendor’s name (Administrator : siemens as another example).To make matters worse, password lockout policies on operational technology is usually lax. The reasoning for this is to prevent accidental lockouts from causing issues with the process, which is generally a good idea. However, this provides attackers a lot of leeway for brute forcing credentials. This issue in combination with the prevalence of default, weak, or shared credentials makes password guessing attacks extremely viable for gaining access to the process control network.
- Process Control Network Credentials Stored in Plaintext – Credentials for operational technology are often included in documentation. Any repositories for process control network documentation should have been enumerated as part of the information gathering phase of a segmentation assessment. Any documentation found in these repositories should be searched for passwords that can be used on entry-points.
- Corporate Network Domain is Extended into Process Control Network – In cases where there are dual-homed hosts or workstations with VPN access on the corporate network, it is fairly common for these hosts to be joined to the corporate network’s Active Directory domain. Because of this, it is often possible to be able to access these hosts directly using highly-privileged Active Directory accounts from the corporate network such as Domain Administrators. More security-minded organizations may have taken the extra step of limiting access to these hosts to only specific users or groups. However, having compromised the corporate network, it is a relatively simple task to locate users of interest on the corporate network and extract their passwords from memory.
- Exposed Vulnerable Services – Entry-points can often have additional services such as databases or web applications that can be vulnerable to common attacks. Any hosts with access to the process control network should be scanned thoroughly to identify exposed services. These services should then be probed to find new vulnerabilities or researched to find publicly-available vulnerabilities or common misconfigurations that could lead to compromising the host.
Easy wins/should not ever happen
If you are ever conducting a pentest against ICS infrastructure, the following are findings that should immediately be written up as critical:
- Modbus/DNP3 traffic on corporate network
- HMIs accessible from the corporate network as not read-only
- NMAPing a nuclear reactor causes a meltdown.
Hopefully, that last scenario NEVER happens, but if it does, write it up and move on.
Conclusion
Despite the fact that the world around us depends upon ICS technologies, interacting with and securing them still has a large gap in knowledge for both the public and the information security community. It is our hope that these blog posts have proved insightful and can help move the conversation forward.
Author: Jordan Parkin
©Aon plc 2023
