Perls of Wisdom: Responding to NetScaler CVE-2019-19781

Background

On December 17, 2019, Citrix announced a vulnerability within the Citrix Application Delivery Controller (Citrix ADC) and Citrix Gateway devices affecting versions 13.0, 12.1, 12.0, 11.1, and 10.5 that was assigned CVE identifier CVE-2019-19781. Citrix did not immediately release a software patch or other mitigation for this vulnerability. The affected devices are also commonly referred to as “NetScaler” devices due to the legacy name prior to the Citrix acquisition of the NetScaler Company. Citrix ADC devices are load balancers, which means they distribute connections to applications across many servers, and thus are typically placed at the perimeter of networks.

On Friday, January 10, 2020, a group of security researchers code-named “Project Zero India” publicly released code on GitHub to exploit this Citrix vulnerability, followed by the release of other public exploits. The exploit worked in two stages — stage one leveraged a vulnerable Perl script to create specially crafted XML files and stage two used a directory traversal technique to invoke code stored in those XML files.

Citrix ADCs use application templates and bookmarks in the XML format to define URLs available for users on login, such as links to documentation, help desk sites, and applications (remote desktop or servers a user may have access to). The XML files uploaded by attackers take advantage of the URL and application delivery capability to introduce undesired remote code execution (“RCE”) on Citrix ADC devices.

Software patches were made available for Citrix ADC devices on January 19, 2020, but there was a one month period prior to the patch release when devices were unprotected unless other mitigations were implemented, such as a Web Application Firewall or requiring users to VPN prior to accessing the gateway.

Triage and Observations

As soon as public exploits were available, the compromise of Citrix ADC devices was widespread. Many of the same indicators of compromise were seen on honeypots reported to the SANS Internet Storm Center related to cryptocurrency miner software and web shells written in Perl. Many of these public indicators of compromise, especially related to cron jobs for the “nobody” user, IP addresses related to the crypto miners, and malware dubbed “NOTROBIN” deployed by an unknown threat group were used to create a triage script released by FireEye in conjunction with Citrix. The script will find the most common exploit indicators and log entries demonstrating attempts to perform directory traversal. However, a Citrix ADC device should not be considered clean if the script does not identify any of the known indicators of compromise, as caveated on FireEye’s blog releasing the script. Logs, XML files, or other artifacts may have been deleted by the time of triage, therefore being unavailable for review by a script that cannot check deleted space. Payloads have been found in less common paths, which the triage script does not check and there should be considerations that a more sophisticated attacker could attempt to evade detection and analysis through some other means. Given this, it would be wise to have a more thorough forensic analysis conducted.

Almost all Citrix ADC devices we have reviewed contained XML template files with malicious code to export configuration and local credential files, which attackers could use at a later time. Stroz Friedberg Incident Response Services recommends changing all administration credentials on affected devices, since many of the payloads delivered by attackers could end up deleted and unavailable for review. The highlighted portion in Figure 1 below reveals template code used to output the “/etc/passwd” file, and is one example of many where attackers have used the exploit to gather sensitive information.

Figure 1: Export Local Linux “passwd” File

We also identified PHP web shells in locations not included in the triage script.  The XML template files used to create the PHP web shell was no longer present on the active file system, and the code was recovered from deleted space. The code in the XML was obfuscated with layers of URI and base 64 encoding, as shown in Figure 2 below with the first layer of URI encoded data for the template.

Figure 2: XML Template File Encoded with PHP Web Shell Delivery

Once decoded, the XML file delivers a simple PHP web shell to “/var/vpn/themes/admin.php” which would execute commands delivered via any POST request to that path, as shown in Figure 3 below:

Figure 3: Decoded PHP Web Shell

This type of backdoor could be used any time an attacker returned to a system, and would not be found without thorough investigation.  Though simple, the capabilities of this web shell are only limited to the software installed on the Citrix ADC server, and could be used to load additional toolkits for lateral movement to other systems.  Response to any Citrix ADC compromise should include review of any available logs for SMB scanning which might indicate attempts to use Eternal Blue exploits, or other traffic from the Citrix ADC device that would be outside of the normal course of business.

Another novel exploit discovered through review of affected Citrix ADC devices is the use of “nshttp_profile_ids” files, which can load data for a user profile, as illustrated in Figure 4 below.

Figure 4: “nshttp_profile_ids” File with URI and Hex Encoded Perl

The data in Figure 4 decodes into a simple Perl listener as shown in Figure 5 below, but this technique could be leveraged for more nefarious payloads:

Figure 5: Decoded Perl Listener

Initial Response and Preservation

Information technology personnel are often quick to isolate affected systems, and sometimes even shut them down during incident response.  In most instances, volatile data such as active network connections will be lost if the server is shut down prior to data being preserved.  For Citrix ADC devices, be aware that in most instances, the root partition is often backed by a BSD memory disk, meaning that critical configuration files in “/etc,” and potentially malicious XML files in “/netscaler/portal/scripts” and “/netscaler/portal/templates,” are stored in memory and could be lost.  Figure 6 below shows the typical mount points for a Citrix ADC, and the highlighted portion illustrates the root partition mounted from a location in running memory, “dev/md0.”

Figure 6: Common Citrix ADC File System Layout

This configuration is particularly impactful on virtual machine and cloud deployments like Amazon Web Services (“AWS”), where initial responders often take snapshots or clones for preservation purposes and then immediately rebuild machines.  On its face, this approach seems completely appropriate and consistent with best practices (i.e., preserve the current state, remove the affected system from the environment, and deploy a clean replacement to prevent a business interruption).  However, in most cases, valuable data can be lost as a result of this approach because the data loaded from memory for the root filesystem is not persistent upon reboot.  Modifying the response to include capturing a live image (and running memory) can make these data available for review and have a significant impact on the investigation.  The logs for system activity, such as commands executed and http access logs, and other forensic artifacts as described in the United States CERT article can be found on the disk image, as they are in the partition mounted at “/var” showing as a physical device in Figure 6 above.

Another preservation approach commonly used – particularly on *nix systems — is to create a collection of relevant files using zip or tar in a way that also preserves filesystem metadata such as timestamps.  This can be done using command line tools, but can be cumbersome to limit, and would also not provide some data structures that could be valuable during an investigation.  We recommend creating an image of the physical disk (/dev/sd0), as well as a live image of the memory device (/dev/md0) over SSH.  The default shell for the “nsroot” user is the NetScaler shell (“nssh”), instead of the usual tcsh or Bash, and thus “shell” must be prepended to any shell commands.  Figure 7 below shows a command to take a live image over SSH utilizing both the dd and dcfldd (a fork of dd with additional features useful for forensic preservation) software:

Figure 7: Command to Collect Disk Image of Root Filesystem over SSH on Citrix ADC

One consideration when working in AWS is the licensing issues inherent with instances launched from the Amazon Marketplace.  A snapshot of an AWS instance cannot be mounted to any instance other than another Citrix ADC.  Typical *nix commands for preservation will work on Citrix ADC, since it is FreeBSD 8.4 under the hood with some kernel customization.

Recommendations

• For any Citrix ADC devices within the environment,
  • Review for unauthorized access;
  • Isolate from the network until review is complete; and
  • Patch as soon as possible. Patches for CVE-2019-19781 and further information about the vulnerability are available on the Citrix website.
• Citrix ADC devices could be placed behind a VPN solution for further security. Note that there can be issues with the proprietary Citrix Independent Computing Architecture (ICA) protocol used between servers and clients being brokered by some VPN and load balancing solutions. Testing should be done prior to any implementation.
• Other potential solutions to mitigate directory traversal attacks on perimeter devices are to:
  • Place them behind a Web Application Firewall to protect against common attacks. Implementation of a WAF would have the added benefit to any later review of actions taken involving web shells since most commands are executed through “POST” requests, which would be logged by WAF solutions, but otherwise not present in local logs on the Citrix ADC.
  • Implement File Integrity Monitoring solutions to alert on any scripts or other files uploaded to monitored directories or changes to any monitored configuration files.
• For any Citrix ADC devices confirmed compromised, it is recommended that the device is rebuilt and clean configuration restored.
• Refer to the Citrix best practices documentation for redeployment of Citrix ADC devices.

Authors: Erik Iker and Noah Rubin