Citrix Bleed Vulnerability: Background and Recommendations

Updated November 30, 2023

Since our initial post, it has come to our attention that a LockBit affiliate has exploited the CitrixBleed vulnerability to gain initial access. This affiliate acted swiftly and effectively, locating sensitive files and proceeding to exfiltrate them using the rclone.exe tool.

Surprisingly, despite successfully exfiltrating the files, no endpoints or files were encrypted as typically seen in ransomware attacks. Instead, the LockBit affiliate employed a batch script named 1.bat to create a readme file called “!important_read_me!.txt” in every directory of the targeted file server from which they exfiltrated the files.

If a user opened the readme file, they were greeted with a message stating “Hello, this is LockBit,” and including a unique customer ID. The file also provided multiple links to onion sites where the impacted user could initiate negotiations using their customer ID.

Our team of security experts will provide updates if new relevant information becomes available.

Key Points

• Citrix Bleed (CVE-2023-4966) is a critical vulnerability affecting multiple versions of Citrix Netscaler Gateway and ADC products that could enable attackers to retrieve sensitive information and hijack user sessions.
• Exploited as a zero-day vulnerability since summer 2023, at least four threat groups are leveraging Citrix Bleed, with one group automating the attack process. ReliaQuest has observed Citrix Bleed exploitation in multiple customer environments.
• Urgent remedial action, including installing updated versions of Netscaler Gateway and ADC and killing active sessions, is strongly recommended by CISA and Citrix’s owner Cloud Software Group.

Citrix Bleed: What’s Happening

Citrix Bleed (CVE-2023-4966) is a critical vulnerability affecting Citrix Netscaler Gateway and Netscaler ADC products—network devices used for load balancing, firewall implementation, traffic management, virtual private network (VPN), and user authentication. By exploiting this flaw, attackers may be able to retrieve sensitive information (including session authentication cookies) from vulnerable appliances and subsequently hijack a user’s session.

The following supported versions of NetScaler ADC and NetScaler Gateway are affected by the vulnerability:

• NetScaler ADC and NetScaler Gateway 14.1 before 14.1-8.50
• NetScaler ADC and NetScaler Gateway 13.1 before 13.1-49.15
• NetScaler ADC and NetScaler Gateway 13.0 before 13.0-92.19
• NetScaler ADC 13.1-FIPS before 13.1-37.164
• NetScaler ADC 12.1-FIPS before 12.1-55.300
• NetScaler ADC 12.1-NDcPP before 12.1-55.300

NetScaler ADC and NetScaler Gateway version 12.1 are  now End-of-Life (EOL) and is also vulnerable. Customers using Citrix-managed cloud services or Citrix-managed Adaptive Authentication products are not impacted.

Citrix released a patch for this flaw on October 10, 2023, but attackers have been abusing it as a zero-day vulnerability since late August 2023.

On October 25, 2023, researchers released a proof-of-concept (PoC) exploit for Citrix Bleed and noted that the vulnerability stems from a buffer-related flaw in Citrix NetScaler ADC and NetScaler Gateway that allows for sensitive information disclosure when configured as a Gateway (VPN virtual server, ICA Proxy, CVPN, RDP Proxy) or AAA virtual server.

At least four different threat groups are exploiting Citrix Bleed and at least one has automated the attack chain. The US Cybersecurity and Infrastructure Security Agency (CISA) has released guidance for addressing this flaw, including patching vulnerable appliances.

In just two days—November 7 to 9, 2023—ReliaQuest has identified multiple unique customer incidents featuring Citrix Bleed exploitation. Both CISA and Citrix’s owner Cloud Software Group have stressed the urgency of taking remedial action as soon as possible.

Observed Exploitation Activity

ReliaQuest has observed consistent examples of exploitation in the wild for Citrix Bleed that have produced similar signals that can be utilized for detection purposes. Utilizing the Citrix ADC Syslog logging capability, the “TCPCONNSTAT” message under the “SSLVPN” function can uncover potential indicators of threat actors having exploited the vulnerability to hijack a user’s session.

Within the TCPCONNSTAT message, a mismatch between the fields “Client IP” and “Source” can be an indicator of a threat actor actively utilizing a previously authenticated session.

Use the following sigma rule to detect a concurrent Citrix user session from different IPs.

Sigma Rule

title: Concurrent Citrix User Session From Multiple IPs
description: This rule detects a potential exploitation of Citrix Bleed (CVE-2023-4966).
tags:
- attack.initial_access
- attack.t1190
logsource:
product: Citrix Netscaler
detection:
selection:
function: "SSLVPN"
message: "TCPCONNSTAT"
condition: selection and Client IP does not equal Source
fields:
- Client_ip
- User
- SessionId
- Source
falsepositives:
- Legitimate causes such as changing networks, cycling on/off VPN 
as well as DHCP can cause Client IPs to change during a session.
level: high

Note: While this can be a strong indicator for potential abuse of this vulnerability, there are certain caveats to consider.

• Without X-Forwarded-For support, it can be difficult to determine the true IP address.
• There are legitimate causes that can change Client IPs during a session, such as changing networks or cycling on/off VPN as well as DHCP.
• To fulfill this use case, it is important to prioritize readiness by excluding specific IP address ranges used in NAT configurations by your organization. In this scenario, client IP addresses are logged without attempting to identify the individual clients behind them.

Observed Post-Exploitation Activity

ReliaQuest has identified multiple cases in customer environments in which threat actors have used the Citrix Bleed exploit. Having gained initial access, the adversaries quickly enumerated the environment, with a focus on speed over stealth. In some cases, the attackers moved quickly to identify and exfiltrate data shares; in other scenarios, the threat actor may have attempted to deploy ransomware. The following list enumerates MITRE ATT&CK techniques we have observed in one or more incidents.

Note: Although these are the most common techniques we have observed, additional techniques have triggered ReliaQuest detections. ReliaQuest has provided customers with a comprehensive set of detection rules both via email and within GreyMatter.

Discovery

• T1482 – Domain Trust Discovery: Following exploitation, ADFind was used to gather information on organizational units (OUs) and domain trusts in Active Directory.
• T1018 – Remote System Discovery: As is typical following initial access discovery, commands such as nltest/dclist were used to enumerate domain controllers in the environment.

Lateral Movement

• T1021.001 – Remote Desktop Protocol: In most cases, we saw attackers use RDP to move laterally to high-value servers following initial access.

Exfiltration

• T1567.002 – Exfiltration Over Web Service: Exfiltration to Cloud Storage: Attackers were observed utilizing Rclone, a command line program to manage files on cloud storage, to exfiltrate data to a MEGA cloud storage site.

Command and Control

• T1090.002 – Proxy: External Proxy: In addition to using tools such as Remote Monitoring and Management (RMM) software and command-and-control (C2) frameworks such as Cobalt Strike, attackers were seen using proxy tools like SystemBC to establish additional concealed communication channels.

Credential Access

• T1558.003 – Steal or Forge Kerberos Tickets: Kerberoasting: To get credentials for privileged accounts and move laterally through the environment, adversaries were seen using Rubeus to execute KerberosRequestorSecurityToken.GetRequest to request tickets for service accounts which could then be cracked offline.
• T1003.003 – OS Credential Dumping: NTDS: After gaining access to a Domain Controller, attackers used ntdsutil to dump the Active Directory database file “NTDS.dit” allowing them access to all Active Directory user hashes for offline cracking.

Privilege Escalation

• T1068 – Exploitation for Privilege Escalation: In multiple cases, attackers were seen using tools such as Invoke-ZeroLogon.ps1 and SharpZeroLogon in an attempt to exploit CVE-2020-1472, also known as ZeroLogon, to quickly escalate privileges to Domain Admin.

Forecast

Security researchers have reported multiple cases of adversaries exploiting the Citrix Bleed vulnerability in the wild. Researchers estimate around 20,000 instances of Citrix devices have had session tokens stolen. Querying the internet server search service Shodan indicates that approximately 18,000 ADC appliances are still publicly available. With the release of a PoC, mass exploitation is highly likely to occur, affecting organizations that have not implemented the advised measures.

Recommendations and Best Practices

Update Guidance

CISA and Cloud Software Group urge organizations running affected builds that have configured customer-managed NetScaler ADC as a gateway (VPN virtual server, ICA proxy, CVPN, RDP proxy) or as an AAA virtual server to install the following updated versions of NetScaler ADC and NetScaler Gateway as soon as possible:

• NetScaler ADC and NetScaler Gateway 14.1-8.50  and later releases
• NetScaler ADC and NetScaler Gateway  13.1-49.15  and later releases of 13.1
• NetScaler ADC and NetScaler Gateway 13.0-92.19 and later releases of 13.0
• NetScaler ADC 13.1-FIPS 13.1-37.164 and later releases of 13.1-FIPS
• NetScaler ADC 12.1-FIPS 12.1-55.300 and later releases of 12.1-FIPS
• NetScaler ADC 12.1-NDcPP 12.1-55.300 and later releases of 12.1-NDcPP

NetScaler ADC and NetScaler Gateway version 12.1 is now End-of-Life (EOL); CISA recommends upgrading appliances to one of the supported versions that address the vulnerabilities.
Organizations using NetScaler ADC or NetScaler Gateway instances on SDX hardware must upgrade VPX instances (the underlying SDX hardware itself is not affected).
NetScaler ADC and NetScaler Gateway appliances that are not configured as a gateway (VPN virtual server, ICA proxy, CVPN, or RDP proxy) or as an AAA virtual server (traditional load balancing configurations, for example) and related products such as NetScaler Application Delivery Management (ADM) and Citrix SD-WAN are not affected.

Killing Active Sessions

Cloud Software Group recommends killing all active and persistent sessions using the following commands:

• kill icaconnection -all
• kill rdp connection -all
• kill pcoipConnection -all
• kill aaa session -all
• clear lb persistentSessions

What ReliaQuest Is Doing

We have shared relevant detection information with our customers via email and within GreyMatter. Customers can reach out to their customer success managers for additional information.

We will also continue to update this blog as new information arises.