“This vulnerability is now under mass exploitation.” Citrix Bleed bug bites hard

A vulnerability that permits attackers to bypass multifactor authentication and entry enterprise networks utilizing {hardware} offered by Citrix is beneath mass exploitation by ransomware hackers regardless of a patch being out there for 3 weeks.

Citrix Bleed, the frequent identify for the vulnerability, carries a severity score of 9.4 out of a potential 10, a comparatively excessive designation for a mere information-disclosure bug. The rationale: the data disclosed can embody session tokens, which the {hardware} assigns to units which have already efficiently supplied credentials, together with these offering MFA. The vulnerability, tracked as CVE-2023-4966 and residing in Citrix’s NetScaler Software Supply Controller and NetScaler Gateway, has been beneath energetic exploitation since August. Citrix issued a patch on October 10.

Repeat: This isn’t a drill

Assaults have solely ramped up just lately, prompting safety researcher Kevin Beaumont on Saturday to declare: “This vulnerability is now beneath mass exploitation.” He went on to say, “From speaking to a number of organizations, they’re seeing widespread exploitation.”

He stated that as of Saturday, he had discovered an estimated 20,000 cases of exploited Citrix units the place session tokens had been stolen. He stated his estimate was based mostly on working a honeypot of servers that masquerade as susceptible Netscaler units to trace opportunistic assaults on the Web. Beaumont then in contrast these outcomes with different information, together with some supplied by Netflow and the Shodan search engine.

In the meantime, GreyNoise, a safety firm that additionally deploys honeypots, was displaying exploits for CVE-2023-4966 coming from 135 IP addresses when this submit went stay on Ars. That’s a 27-fold enhance from the 5 IPs noticed GreyNoise noticed 5 days in the past.

The latest numbers out there from safety group Shadowserver confirmed that there have been roughly 5,500 unpatched devices. Beaumont has acknowledged that the estimate is at odds together with his estimate of 20,000 compromised units. It’s not instantly clear what should be blamed for the discrepancy.

The vulnerability is comparatively straightforward for knowledgeable folks to use. A easy reverse-engineering of the patch Citrix launched exhibits the capabilities which are susceptible, and from there, it’s not arduous to write down code that exploits them. Making assaults even simpler, a handful of proof-of-concept exploits can be found on-line.

In a detailed technical analysis, researchers from Assetnote wrote:

We discovered two capabilities that stood out ns_aaa_oauth_send_openid_config and ns_aaa_oauthrp_send_openid_config. Each capabilities carry out an identical operation, they implement the OpenID Join Discovery endpoint. The capabilities are each accessible unauthenticated by way of the /oauth/idp/.well-known/openid-configuration and /oauth/rp/.well-known/openid-configuration endpoints respectively.

Each capabilities additionally included the identical patch, an extra bounds examine earlier than sending the response. This may be seen within the snippets beneath displaying the earlier than and after for ns_aaa_oauth_send_openid_config.

Authentic

iVar3 = snprintf(print_temp_rule,0x20000,
           	"{"issuer": "https://%.*s", "authorization_endpoint": "https://%.*s/oauth/ idp/login", "token_endpoint": "https://%.*s/oauth/idp/token", "jwks_uri":  "https://%.*s/oauth/idp/certs", "response_types_supported": ["code", "toke n", "id_token"], "id_token_signing_alg_values_supported": ["RS256"], "finish _session_endpoint": "https://%.*s/oauth/idp/logout", "frontchannel_logout_sup ported": true, "scopes_supported": ["openid", "ctxs_cc"], "claims_support ed": ["sub", "iss", "aud", "exp", "iat", "auth_time", "acr", "amr ", "email", "given_name", "family_name", "nickname"], "userinfo_endpoin t": "https://%.*s/oauth/idp/userinfo", "subject_types_supported": ["public"]}"
           	,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8);
authv2_json_resp = 1;
iVar3 = ns_vpn_send_response(param_1,0x100040,print_temp_rule,iVar3);

Patched

uVar7 = snprintf(print_temp_rule,0x20000,
           	"{"issuer": "https://%.*s", "authorization_endpoint": "https://%.*s/oauth/ idp/login", "token_endpoint": "https://%.*s/oauth/idp/token", "jwks_uri":  "https://%.*s/oauth/idp/certs", "response_types_supported": ["code", "toke n", "id_token"], "id_token_signing_alg_values_supported": ["RS256"], "finish _session_endpoint": "https://%.*s/oauth/idp/logout", "frontchannel_logout_sup ported": true, "scopes_supported": ["openid", "ctxs_cc"], "claims_support ed": ["sub", "iss", "aud", "exp", "iat", "auth_time", "acr", "amr ", "email", "given_name", "family_name", "nickname"], "userinfo_endpoin t": "https://%.*s/oauth/idp/userinfo", "subject_types_supported": ["public"]}"
           	,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8,uVar5,pbVar8);
uVar4 = 0x20;
if (uVar7 < 0x20000) {
	authv2_json_resp = 1;
	iVar3 = ns_vpn_send_response(param_1,0x100040,print_temp_rule,uVar7);
	...
}

The operate is fairly easy, it generates a JSON payload for the OpenID configuration and makes use of snprintf to insert the system’s hostname on the acceptable areas within the payload. Within the authentic model, the response is shipped instantly. Within the patched model, the response is barely despatched if snprintf returns a worth lower than 0x20000.

The vulnerability happens as a result of the return worth of snprintf is used to find out what number of bytes are despatched to the shopper by ns_vpn_send_response. It is a downside as a result of snprintf doesn’t return what number of bytes it did write to the buffer, snprintf returns what number of bytes it would have written to the buffer if the buffer was sufficiently big.

To take advantage of this, all we wanted to do was work out how you can get the response to exceed the buffer measurement of 0x20000 bytes. The appliance would then reply with the utterly crammed buffer, plus no matter reminiscence instantly adopted the print_temp_rule buffer.

‍Exploiting the Endpoint

Initially we thought the endpoint would in all probability not be exploitable. The one information that was inserted was the hostname, which is one thing that wanted administrator entry to configure. Fortunately for us, we had been mistaken and the worth inserted into the payload didn’t come from the configured hostname. It truly got here from the HTTP Host header.

We had been additionally lucky that NetScaler inserts the hostname into the payload six occasions, as this meant we might hit the buffer restrict of 0x20000 bytes with out working into points as a result of both the Host header or the entire request was too lengthy.

We put collectively the next request and despatched it to our NetScaler occasion.

GET /oauth/idp/.well-known/openid-configuration HTTP/1.1
Host: a <repeated 24812 occasions>
Connection: shut

We obtained the response proven beneath with the non-printable characters eliminated.

HTTP/1.1 200 OK
X-Content material-Kind-Choices: nosniff
X-XSS-Safety: 1; mode=block
Content material-Size: 147441
Cache-control: no-cache, no-store, must-revalidate
Pragma: no-cache
Content material-Kind: software/json; charset=utf-8
X-Citrix-Software: Receiver for Net

{"issuer": "https://aaaaa ...<omitted>... aaaaaaaaaaaaaaaaí§¡
ð
í§¡-ª¼tÙÌåDx013.1.48.47à
d98cd79972b2637450836d4009793b100c3a01f2245525d5f4f58455e445a4a42HTTP/1.1 200 OK
Content material-Size: @@@@@
Encode:@@@
Cache-control: no-cache
Pragma: no-cache
Content material-Kind: textual content/html
Set-Cookie: NSC_AAAC=@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@;Safe;HttpOnly;Path=/

{"classes":[],"assets":[],"subscriptionsEnabled":false,"username":null}
ð
å
å
PÏÏ
H¡
éÒÏ
eGÁ"RDEFAULT
ò #pack200-gzip
compressdeflategzip
dentity
þÿÿÿÿÿ
©VPN_GLOBALÿÿÿÿÿÿ   è"AAA_PARAMí

We might clearly see a number of leaked reminiscence instantly following the JSON payload. Whereas a number of it was null bytes, there was some suspicious trying data within the response.

The identify Citrix Bleed is an allusion to Heartbleed, a special vital data disclosure vulnerability that turned the Internet on its head in 2014. That vulnerability, which resided within the OpenSSL code library, got here beneath mass exploitation and allowed the pilfering of passwords, encryption keys, banking credentials, and all types of different delicate data. Citrix Bleed isn’t as dire as a result of there are fewer susceptible units in use.

However Citrix Bleed remains to be loads dangerous. Organizations ought to think about all Netscaler units to have been compromised. This implies patching any remaining unpatched units. Then, all credentials ought to be rotated to make sure any session tokens which may have been leaked are invalidated. Final, organizations ought to examine their units and infrastructure for indicators of compromise. Safety agency Mandiant has in-depth safety steering here.