reader comments
16 with
A vulnerability that allows attackers to bypass multifactor authentication and access enterprise networks using hardware sold by Citrix is under mass exploitation by ransomware hackers despite a patch being available for three weeks.
Citrix Bleed, the common name for the vulnerability, carries a severity rating of 9.4 out of a possible 10, a relatively high designation for a mere information-disclosure bug. The reason: the information disclosed can include session tokens, which the hardware assigns to devices that have already successfully provided credentials, including those providing MFA. The vulnerability, tracked as CVE-2023-4966 and residing in Citrix’s NetScaler Application Delivery Controller and NetScaler Gateway, has been under active exploitation since August. Citrix issued a patch on October 10.
Repeat: This is not a drill
Attacks have only ramped up recently, prompting security researcher Kevin Beaumont on Saturday to declare: “This vulnerability is now under mass exploitation.” He went on to say, “From talking to multiple organizations, they are seeing widespread exploitation.”
He said that as of Saturday, he had found an estimated 20,000 instances of exploited Citrix devices where session tokens had been stolen. He said his estimate was based on running a honeypot of servers that masquerade as vulnerable Netscaler devices to track opportunistic attacks on the Internet. Beaumont then compared those results with other data, including some provided by Netflow and the Shodan search engine.
Meanwhile, GreyNoise, a security company that also deploys honeypots, was showing exploits for CVE-2023-4966 coming from 135 IP addresses when this post went live on Ars. That’s a 27-fold increase from the five IPs spotted GreyNoise saw five days ago.
roughly 5,500 unpatched devices. Beaumont has acknowledged that the estimate is at odds with his estimate of 20,000 compromised devices. It’s not immediately clear what was causing the discrepancy.
The vulnerability is relatively easy for experienced people to exploit. A simple reverse-engineering of the patch Citrix released shows the functions that are vulnerable, and from there, it’s not hard to write code that exploits them. Making attacks even easier, a handful of proof-of-concept exploits are available online.
In a detailed technical analysis, researchers from Assetnote wrote:
We found two functions that stood out
ns_aaa_oauth_send_openid_configandns_aaa_oauthrp_send_openid_config. Both functions perform a similar operation, they implement the OpenID Connect Discovery endpoint. The functions are both accessible unauthenticated via the/oauth/idp/.well-known/openid-configurationand/oauth/rp/.well-known/openid-configurationendpoints respectively.Both functions also included the same patch, an additional bounds check before sending the response. This can be seen in the snippets below showing the before and after for
ns_aaa_oauth_send_openid_config.Original
Patched
The function is pretty simple, it generates a JSON payload for the OpenID configuration and uses
snprintfto insert the device’s hostname at the appropriate locations in the payload. In the original version, the response is sent immediately. In the patched version, the response is only sent ifsnprintfreturns a value less than0x20000.The vulnerability occurs because the return value of
snprintfis used to determine how many bytes are sent to the client byns_vpn_send_response. This is a problem becausesnprintfdoes not return how many bytes it did write to the buffer,snprintfreturns how many bytes it would have written to the buffer if the buffer was big enough.To exploit this, all we needed to do was figure out how to get the response to exceed the buffer size of
0x20000bytes. The application would then respond with the completely filled buffer, plus whatever memory immediately followed theprint_temp_rulebuffer.Exploiting the Endpoint
Initially we thought the endpoint would probably not be exploitable. The only data that was inserted was the hostname, which is something that needed administrator access to configure. Luckily for us, we were wrong and the value inserted into the payload did not come from the configured hostname. It actually came from the HTTP
Hostheader.We were also fortunate that NetScaler inserts the hostname into the payload six times, as this meant we could hit the buffer limit of
0x20000bytes without running into issues because either theHostheader or the whole request was too long.We put together the following request and sent it to our NetScaler instance.
We received the response shown below with the non-printable characters removed.
We could clearly see a lot of leaked memory immediately following the JSON payload. While a lot of it was null bytes, there was some suspicious looking information in the response.
turned the Internet on its head in 2014. That vulnerability, which resided in the OpenSSL code library, came under mass exploitation and allowed the pilfering of passwords, encryption keys, banking credentials, and all kinds of other sensitive information. Citrix Bleed isn’t as dire because there are fewer vulnerable devices in use.
But Citrix Bleed is still plenty bad. Organizations should consider all Netscaler devices to have been compromised. This means patching any remaining unpatched devices. Then, all credentials should be rotated to ensure any session tokens that might have been leaked are invalidated. Last, organizations should inspect their devices and infrastructure for signs of compromise. Security firm Mandiant has in-depth security guidance here.
