Burning Zero Days: Suspected Nation-State Adversary Targets Ivanti CSA

Affected Platforms: Ivanti Cloud Services Appliance version 4.6 and prior
Impacted Users: Any organization
Impact: Remote attackers gain control of the vulnerable systems
Severity Level: Critical

Today FortiGuard Labs is releasing this blog post about a case where an advanced adversary was observed exploiting three vulnerabilities affecting the Ivanti Cloud Services Appliance (CSA). At the time of our investigation, two out of the three identified vulnerabilities were not publicly known. This incident is a prime example of how threat actors chain zero-day vulnerabilities to gain initial access to a victim’s network.

Background

In a recent incident response engagement, FortiGuard Incident Response (FGIR) services were engaged by a customer to investigate malicious communication originating from their network. During the investigation, FGIR came across an adversary who had gained access to the customer’s network by exploiting the CVE-2024-8190 and two previously unknown vulnerabilities affecting the PHP front end of the Ivanti CSA appliance.

The incident was detected by the customer on September 9, 2024, when some of its internal systems were found to be communicating to a malicious IP address, 206[.]189[.]156[.]69. FGIR was engaged the next day.

Vulnerabilities Overview and Disclosure

During the IR investigation, FGIR observed that the threat actor exploited the vulnerability CVE-2024-8190 in conjunction with the following two previously publicly unknown vulnerabilities:

• A publicly unknown path traversal vulnerability on the resource /client/index.php, to gain unauthorized access to other resources like users.php, reports.php etc. (CVE-2024-8963, disclosed September 19)
• A publicly unknown command injection vulnerability affecting the resource reports.php. (CVE-2024-9380, disclosed October 8)
  

These resources are located under the root folder of the PHP web front, which serves as the management console of the CSA.

On September 19, 2024, FGIR disclosed to Ivanti’s security team the discovery of the two new vulnerabilities. During the meeting, the Ivanti team claimed that they were aware and tracking the two publicly unknown exploited vulnerabilities.

On September 19, Ivanti published the advisory for CVE-2024-8963, which addressed the path traversal vulnerability.

Vulnerabilities Details

On September 10, 2024, at 14:00:02, Ivanti published the security advisory CVE-2024-8190 on their forum. The advisory informed about the discovery of an authenticated command injection vulnerability in the DateTimeTab.php resource, affecting CSA 4.6 with patch 518 and earlier versions.

On September 13, 2024, the CVE-2024-8190 vulnerability was added to the CISA’s Known Exploited Vulnerabilities list. On the same date, Ivanti updated their security advisory to mention that, following public disclosure of the September 10^th, exploitation of the command injection vulnerability had been observed in the wild.

On September 16, 2024, the research team at Horizon3.ai published the details related to the CVE-2024-8190 vulnerability and also released a proof of concept exploit code.

Path Traversal Vulnerability – /client/index.php

During the incident response investigation, FGIR observed that the threat actor exploited a path traversal vulnerability on the resource /client/index.php to gain unauthorized authenticated access to the resource /gsb/users.php by sending the following web request:

The first of such requests was sent by the threat actor on September 4, 2024, at 06:53:14 UTC, right before the exploitation of the command injection vulnerability, affecting the resource /gsb/reports.php.

The resource /client/index.php on the PHP web front of the Ivanti CSA appliance can be accessed by unauthenticated users to download the “LANDESK Remote Assistance Client” software package.

The following picture shows how the resource /client/index.php looks when opened in a browser:

Upon inspecting the /client/index.php’s code, FGIR discovered that, by clicking the “Install now” button present on it, the user is redirected to a resource called /client/download.php:

The resource /client/download.php redirects the user to the page OnDemand.php via the header function.

The resource /client/OnDemand.php contains the code to open a local file called LDSupport.exe, using the php function popen. The local file is served to the user via the php echo command.

The threat actor sent a malformed URL to the resource /client/index.php, by inserting %3F.php at the end of the URI, and appended the URL with the location of the php resource to be accessed through path traversal. Using this technique, the threat actor managed to access the resource /gsb/users.php.

The appended resource, /gsb/users.php, was assigned to the variable $filename in the /client/OnDemand.php code, which led to the path traversal vulnerability, allowing the threat actor to view the list of users configured in the CSA appliance. FGIR simulated the exploitation of this vulnerability in its lab environment to understand what information could be acquired with it and the figure below shows the resulting output, which includes the list of users configured on the test appliance:

The threat actor exploited this vulnerability several times over the course of their intrusion to access other resources as well, with connections originating from various IP addresses. As seen in the screenshot below, which has all times expressed using the timezone UTC-007, the threat actor used the same vulnerability to access the resource /gsb/datetime.php as well.

FGIR states with medium confidence that the threat actor exploited this path traversal vulnerability to gain access to the resource /gsb/users.php not only to list users, but also to attempt to create rogue users and gain authenticated access to the CSA web front end.

The messages logs contain evidence of the threat actor creating two users: aiadmin and services, using the CSA utility called dbtool. This was likely performed to maintain persistent, authenticated access to the CSA management console.

CVE-2024-8190 Vulnerability Exploitation - /gsb/DateTimeTab.php

After the threat actor exploited the path traversal vulnerability and enumerated users configured on the CSA appliance, they exploited CVE-2024-8190, the command injection vulnerability affecting the resource /gsb/DateTimeTab.php, to attempt to access the credentials of those users.

FGIR observed evidence of this exploitation in Ivanti’s broker logs, as seen in the snippet below. FGIR has high confidence that the threat actor exploited this vulnerability to gain access to the user, admin’s credentials and use these privileged credentials to carry out the authenticated exploitation of the command injection vulnerability in /clients/reports.php resource.

FGIR found the text “rewritten with new timezone” within the function setPhpTimeZone($timezoneinfo) of the resource DateTimeTab.php.

From the above systemLog function call, it is clear that the value of the variable, $timezone was the one that was manipulated by the threat actor. Tracing back the value of this variable shows that it was deduced from the parameter $timezoneinfo, which was passed to the function setPhpTimeZone.

Tracing back where the setPhpTimeZone was called, FGIR found that this happened within the function handleDateTimeSubmit(&$msg). This function is responsible for handling change submissions when the “Save” button is clicked, as depicted in the screenshot below.

The value of a global variable, $TIMEZONE, which was passed as a variable to the function setPhpTimeZone, was received from the POST request variable TIMEZONE. This implies that the threat actor crafted and sent a malicious POST request to DateTimeTab.php, with the malicious command passed in the POST request variable TIMEZONE.

Going back to the malicious command injected by the threat actor, FGIR decoded the base64 blob, which resulted in the following Python code:

The script extracts the password of the user gsbadmin from the file /opt/landesk/broker/broker.conf and assigns its value to the environment variable PGPASSWORD.

The code targets the latest backup file in the /backups directory and iterates through this directory to find the latest backup file. If it finds one, then it connects to the Postgres database using the gsbadmin credentials, extracts the password of the user admin from the table user_info, and assigns it to the environment variable PGPASSWORD.

The script then decompresses the latest backup file, and then iterates through the files to search for a filename that satisfies the regular expression php\w{6}, basically looking for a filename containing the string php followed by six characters (letters or numbers only). In case it finds such a file, it changes the permissions of the file so that everyone has the right to read, write, and execute it. The script then reads the private key from the backup file of the user root, encodes it using base64, and then assigns the resultant value to the column organization in the Postgres database table, user_info.

While assessing the backend Postgres database logs, FGIR found confirmation of successful exploitation of this command injection vulnerability, as seen in the screenshot below:

Command Injection Vulnerability – /gsb/reports.php

Once the threat actor extracted the credentials of the users gsbadmin, admin, and the private key of the user root, they used the stolen credentials to perform authenticated exploitation of the publicly unknown command injection vulnerability affecting the resource /gsb/reports.php. Some of the threat actor requests can be seen in the messages and audit logs below:

Command injection was found to be exploited in the following format, where a php script /subin/tripwire was executed with the parameter –update, followed by a semicolon and a malicious command.

The first malicious command injected by the threat actor was used to create a web shell called help.php in the CSA webroot folder under the /gsb directory.

FGIR looked at the code of the resource reports.php and identified the vulnerability in the line highlighted below in yellow. The script accepts the value of the POST parameter TW_ID and passes it as a parameter to the script /subin/tripwire, without sanitization.

The resource /sbin/tripwire is a PHP wrapper for the binary /usr/sbin/tripwire, which is used to create security reports, when initiated through the gsb/reports.php resource. This is a legitimate functionality of the Ivanti CSA portal.

FGIR inspected the tripwire PHP wrapper script and found the function update which passes the command directly to the installed tripwire binary without sanitization. Since the tripwire PHP wrapper runs with sudo privileges, the injected command also runs with elevated privileges.

FGIR acquired the patch for CVE-2024-8190, and while analyzing the functionality of the patch, FGIR assessed that the file, reports.php was not in the patch script, nor was it listed as a file in the patch folder, leading FGIR to conclude that the command injection vulnerability found in the resource /gsb/reports.php was not addressed in that patch.

Threat Actor Patching Vulnerabilities

On September 10, 2024, when the advisory for CVE-2024-8190 was published by Ivanti, the threat actor, still active in the customer’s network, “patched” the command injection vulnerabilities in the resources /gsb/DateTimeTab.php, and /gsb/reports.php, making them unexploitable.

In the past, threat actors have been observed to patch vulnerabilities after having exploited them, and gained foothold into the victim’s network, to stop any other intruder from gaining access to the vulnerable asset(s), and potentially interfering with their attack operations.

In this case, the threat actor downloaded the patched version of the two vulnerable resources from temp[.]sh and saved them as /tmp/1 on disk, before moving them to the webroot folder and overwriting the vulnerable version of the files with them. Below are the relevant commands:

The modified timestamps of the resources reports.php, and DateTimeTab.php were September 10, 2024, at 12:37:23 UTC and 13:06:10 UTC, respectively, as seen in the screenshots below.

Comparing the original vulnerable version of reports.php to the version patched by the threat actor, shows that the threat actor added a piece of code to replace the semicolon with an underscore in the POST parameter TW_ID, so that command injection using the semicolon is not possible anymore.

FGIR tested the patching in a lab environment and confirmed that the modification by the threat actor does indeed make the resource reports.php unexploitable after the patch. The screenshot below shows the directory testwithoutfix was successfully created by exploiting the command injection vulnerability on the original vulnerable version of reports.php. When the fix is applied to the reports.php file and the command injection is exploited again, the directory testwithfix is not created.

The threat actor also patched the file DateTimeTab.php using the same str_replace function to replace any semicolon in the POST parameter TIMEZONE, with an underscore, rendering the command injection vulnerability using this parameter ineffective.

Other Findings

Lateral Movement

After compromising the internet-facing Ivanti CSA appliance, the threat actor exploited the CVE-2024-29824 SQL Injection vulnerability on Ivanti’s backend SQL database server (SQLS). Sample malicious POST requests exploiting this vulnerability are depicted below.

The threat actor enabled the xp_cmdshell stored procedure via the exploitation and used this stored procedure to attain remote code execution on the SQLS system.

The threat actor created an account called mssqlsvc on the compromised system SQLS and turned off the host firewall.

FGIR also observed that the threat actor ran some basic reconnaissance commands on the SQLS system and attempted to exfiltrate the reconnaissance output using the following PowerShell command:

This is a technique used to exfiltrate data over the DNS protocol. The technique is also known as DNS tunneling. See below for the content of the file that the threat actor tried to exfiltrate:

Threat Actor Commands

FGIR discovered more tactics and techniques used by the threat actor during their intrusion. FGIR extracted and decoded some interesting commands executed by the threat actor from the Linux audit logs and other sources. A sample of those commands are shown in the table below:

Web Shells

During the course of their operations, the threat actor created several webshells. They also modified the legitimate resource, syslog.php, and appended malicious code to it, to use it as a web shell. Some of the web shells found are shown in the table below.

Brute Force Attack

On September 11, at 04:12:00 UTC, the threat actor started an authentication brute force attack against the customer’s internal network assets, using a dictionary attack.

FGIR discovered that the threat actor downloaded a tar file called u from a temp[.]sh URL. This tar file contained three files: brokes, passdic.txt, and u.txt.

The file brokes is a Linux ELF binary, which was used to perform the brute force attack on customer’s network assets. It is likely that brokes used as parameters the list of customer’s users, possibly harvested during a different campaign, in the form of the file u.txt and the password file passdic.txt.

The threat actor downloaded an unknown file called target from temp[.]sh, however this one was not found on the disk.

The threat actor also downloaded a shell script called s.sh, from the temp[.]sh site. This script was used to execute the bruteforce binary brokes and anonymous logins were attempted on LDAP’s port TCP 389 of the attacked assets with several passwords.

ReverseSocks Proxy Tool

During the memory analysis of the CSA appliance, FGIR discovered traces of the use of an open-source go-based proxy tool called ReverseSocks5, which was downloaded and used by the threat actor to perform scanning and brute force attacks on the customer’s internal network, while proxying the traffic through the CSA appliance. The string, which was created in the memory due to an error thrown by the tool, can be seen in the below snippet.

Some other suspicious strings found during the analysis of the memory included some PHP variables found to be populated with suspicious values:

ZjmgmXsB.php was a webshell, which the threat actor was interacting with, while accessing it from the IP address 208[.]105[.]190[.]170.

Root Kit Discovery and Analysis

During the investigation, FGIR discovered that on September 7, 2024, at 03:26:17 UTC, the threat actor attempted to deploy a rootkit in the form of a Linux kernel object (KO) module on the CSA appliance. This attempt was found in the audit logs as seen in the snippet below:

The likely motive behind this was for the threat actor to maintain kernel-level persistence on the CSA device, which may survive even a factory reset. This activity is in line with the public reporting on the compromise of Ivanti CSA appliances, which is available here and here. FGIR decoded the hex string contained in the snippet above and obtained a base64 encoded blob. The initial part of the resultant base64 encoded blob indicates that this is a compressed archive file.

Decompressing the tar file results in the following two files.

The file install.sh is a malicious bash script that installs a malicious kernel object called sysinitd.ko. The compressed archive file was corrupted and only the file install.sh could be retrieved successfully in its entirety, while the file sysinitd was truncated, and the file sysinitd.ko was missing.

FGIR pivoted to the disk image of the CSA appliance and found the sysinitd and sysinitd.ko files in the location /usr/share/empty/init/.

Analysis of Install.sh

The script install.sh was meant to install the malicious rootkit sysinitd.ko on the affected system. The following variables were hard-coded in the script install.sh:

The script starts with the following function call, which reads two parameters:

The first parameter is the INSTALL_NAME string, which is used to rename the two files sysinitd and sysinitd.ko to INSTALL_NAME and INSTALL_NAME.ko respectively.

The second parameter is the path where the script copies the renamed version of sysinitd and sysinitd.ko to. In this case, the threat actor did not supply either of the two parameters. Therefore, the default names sysinitd and sysinitd.ko were used by the script.

The following snippet of code checks if the install path exists and if not, then it creates it:

The script then removes any installed kernel object with the name INSTALL_NAME.ko, using the command rmmod, and then installs the malicious INSTALL_NAME.ko using the command insmod.

The bash script install.sh installs a persistence mechanism using the technique of adding an entry to install the malicious kernel object in the rc.local and rc.d/rc.local files, if the malicious kernel object file is present on disk.

Analysis of the sysinitd and sysinitd.ko

FGIR aims to analyze the rootkit in detail and publish the findings in a follow-up blog post.

Conclusion

The advanced adversaries were observed exploiting and chaining zero-day vulnerabilities to establish beachhead access in the victim’s network. You can read more about the Ivanti CSA zero-day attack in our Threat Signal Report: https://www.fortiguard.com/threat-signal-report/5556.

Fortinet Protections

The malware described in this report is detected and blocked by FortiGuard Antivirus as:

BASH/Agent.030E!tr
ELF/Agent.69A0!tr 
ELF/Agent.7E02!tr 
ELF/Agent.BD!tr

FortiGate, FortiMail, FortiClient, and FortiEDR support the FortiGuard Antivirus service. The FortiGuard antivirus engine is a part of each of those solutions. As a result, customers who have these products with up-to-date protections are protected.

Fortinet has also released the following IPS signatures to protect our customers from the threats contained in the report.

CVE-2024-8190; https://www.fortiguard.com/encyclopedia/ips/56651

The interactsh related URLs are rated as “Malicious Websites” and “Malicious Activities Found” by the FortiGuard Web Filtering service.

FortiGuard IP Reputation and Anti-Botnet Security Service proactively blocks these attacks by aggregating malicious source IP data from the Fortinet distributed network of threat sensors, CERTs, MITRE, cooperative competitors, and other global sources that collaborate to provide up-to-date threat intelligence about hostile sources.

If you believe this or any other cybersecurity threat has impacted your organization, please contact our Global FortiGuard Incident Response Team.

MITRE Mapping

The MITRE ATT&CK framework has been used to refer to the various tactics and techniques used by the threat actor.

IOCs

Network Based Indicators

Host Based Indicators