PART II – perfctl: A Stealthy Malware Targeting Millions of Linux Servers

(I) IoT

4 weeks ago

A Stealthy Malware Targeting Millions of Linux Servers

Read the first part at: https://iotsecuritynews.com/wp-admin/post.php?post=8009&action=edit

Detection of “Perfctl” Malware

To detect Perfctl malware you look for unusual spikes in CPU usage, or system slowdown if the rootkit has been deployed on your server. These may indicate cryptomining activities, especially during idle times.

Monitoring Suspicious System Behavior

Inspect /tmp, /usr, and /root directories for suspicious binaries, especially hidden or masquerading as system files (e.g., perfctl, sh, libpprocps.so, perfcc, libfsnkdev.so). Inspect your /home directory, look for /.local/bin directory with various utilities installed such as ldd, top etc.
Monitor processes for high resource usage, such as binaries like httpd or sh behaving unusually or running from unexpected locations like /tmp.
Check system logs for modifications to ~/.profile, and /etc/ld.so.preload files.

Network Traffic Analysis

Capture network traffic to detect TOR-based communication to external IPs like 80.67.172.162, 176.10.107.180, etc.
Look for outbound connections to cryptomining pools or proxy-jacking services.
Monitor traffic to known malicious hosts or IPs (e.g., 46.101.139.173, 104.183.100.189, and 198.211.126.180).

File and Process Integrity Monitoring

Detect modifications in key system utilities like ldd, top, lsof, and crontab, which might have been replaced with trojanized versions.

Log Analysis

Review logs for unauthorized use of system binaries, presence of suspicious cron jobs, and errors in mesg to detect possible tampering.

Detection of “Perfctl” Malware with Aqua Security

First, we can see some runtime incidents. Below you can see alerts indicating some new binaries were dropped and executed, meaning a drift in our container, in addition to shared object dropped during runtime. These are the additional httpd malware files and the rootkit.

Figure 18: Incidents screen on Aqua Security Platform

We can continue the investigation of this attack by examining the audit logs of these incidents. During this incident there were above 22K audit events, thus we will need to search for specific events, namely, to investigate the attack.

Figure 19: Audit logs screen on Aqua Security Platform

We can filter on specific hosts, containers, enforce groups, cloud resources or even pods. We decided to search for specific events based on the MITRE framework. We used the masquerading technique which is used to describe dropped and executed events. There were 465 incidents, we can now go over all the files that were dropped (or modified) during the attack.

Figure 20: Audit logs screen on Aqua Security Platform

We can learn for instance about the swapping of some binaries with user land rootkits.

Figure 21: Audit logs screen on Aqua Security Platform

You can also learn about inbound traffic or setting up port listening.

Figure 22: Audit logs screen on Aqua Security Platform

Mitigation of “Perfctl” Malware

Patch Vulnerabilities: Ensure that all vulnerabilities are patched. Particularly internet facing applications such as RocketMQ servers and CVE-2021-4043 (Polkit). Keep all software and system libraries up to date.
Restrict File Execution: Set noexec on /tmp, /dev/shm and other writable directories to prevent malware from executing binaries directly from these locations.
Disable Unused Services: Disable any services that aren’t required, particularly those that may expose the system to external attackers, such as HTTP services.
Implement Strict Privilege Management: Restrict root access to critical files and directories. Use Role-Based Access Control (RBAC) to limit what users and processes can access or modify.
Network Segmentation: Isolate critical servers from the internet or use firewalls to restrict outbound communication, especially TOR traffic or connections to cryptomining pools.
Deploy Runtime Protection: Use advanced anti-malware and behavioral detection tools that can detect rootkits, cryptominers, and fileless malware like perfctl.

Appendices

Appendix 1: Initial Access

CVE-2023-33246 is a vulnerability found in RocketMQ, which is a software that manages messages. This vulnerability allows unauthorized execution of commands on systems where RocketMQ is installed. This issue occurs because RocketMQ does not adequately check who is trying to access it, which means anyone, even without permission, can make changes or execute commands. The problem is made worse because the parts of RocketMQ that handle storing and delivering messages were not designed to be directly accessible over the internet, and they don’t require authentication for performing sensitive operations like updating settings. This makes it relatively easy for attackers to exploit this vulnerability.

The initial access was gained via this vulnerability (CVE-2023-33246), led to download and execution of the shell script rconf with the following command:

In Figure 24 below, you can observe the entire rconf script, next we will do a breakdown and explanation of the content.

Figure 24: The rconf script

Appendix 2: Execution Script Analysis

As depicted in Figure 25 below, the script starts with a function that appears to perform a simplified HTTP GET request using a TCP socket directly, mimicking some basic behavior of the curl command. The threat actor is using this, in case the targeted server doesn’t contain curl or wget.

Figure 25: A snippet from the rconf script, illustrating implementation of a HHTP get request command

As you can see in Figure 26 below, the script continues with a simple if condition, that will ensure that the targeted attacked server OS architecture is x86_64. This shows that the threat actor is targeting specific architecture and won’t run on arm for instance.

Figure 26: A snippet from the rconf script, illustrating inspection of the targeted host architecture

Next, the threat actor verifies that the /tmp directory exists and has read, write, and execute permissions. This directory will be used later to store logs, which the malware will update and from which the malware will read instructions or system status.

Figure 27: A snippet from the ‘rconf’ script, illustrating inspection the ‘/tmp’ path

As illustrated in Figure 28 below, the threat actor also verifies that the /tmp directory is mounted with executable permissions. If the noexec option is found in the mount options (no execution permissions), it remounts /tmp with the exec option, allowing execution of binaries from the /tmp directory. This might be necessary for scripts or applications that require executing temporary files stored in /tmp.

Figure 28: A snippet from the ‘rconf’ script, illustrating further inspection of the ‘/tmp’ directory

In addition, the threat actor is creating two directories under /tmp path, which will be used later as auxiliary when running the main payload.

Figure 29: A snippet from the ‘rconf’ script, illustrating creation of directories under the ‘/tmp’ path

Next the threat actor is setting the environment variable A2ZNODE to localhost, if it is not already defined.

Figure 30: A snippet from the ‘rconf’ script, illustrating inspection of the environment variables

In addition, the threat actor is also setting the environment variable VEI to rmq which can stand for vulnerability exploited index to RocketMQ or something similar. Next, this script processes the /tmp/.xdiag/vei file by appending the value of the VEI variable (rmq) to it. If the file /tmp/.xdiag/vei does not exist or is empty, it checks if a secondary file /tmp/.xdiag/vei.1 exists. If it does, the script processes the contents of /tmp/.xdiag/vei, sorts and removes duplicates, and appends the value of VEI. If /tmp/.xdiag/vei.1 does not exist, it directly writes the value of VEI to /tmp/.xdiag/vei. Finally, it unsets the VEI variable

Figure 31: A snippet from the ‘rconf’ script, illustrating preparation of the ‘/tmp’ directory for the malware operation and logging

Finally, this script manages the installation of the main payload by ensuring no other instances are running, downloading the necessary file, and starting a web server. It uses either curl, wget, or the custom download function (mentioned above), verifies the downloaded file, and runs it if valid. The script also includes safeguards to prevent multiple installations from occurring simultaneously. This is important because the initial curl of this script rconf runs iteratively various times throughout the attack.

Figure 32: A snippet from the ‘rconf’ script, illustrating download and installation of the malware under the name ‘httpd’

Now the main payload avatar.php was downloaded, renamed to httpd and executed, we can focus on this binary.

Appendix 3: The main payload (‘httpd’ and ‘sh’) analysis

Analysis of httpd

The binary httpd is a packed ELF (MD5: 656e22c65bf7c04d87b5afbe52b8d800) bears many detections in VirusTotal, including general Linux Trojan, Coinminer, Exploitation tool for CVE-2021-4034, malware dropper etc.

Our analysis shows that in a way all these detections are correct, as in a nutshell this is a multipurpose malware-dropper that contains all the above. Its operation is very interesting as it incorporates dozens of techniques to remain hidden and persistent. Based on our analysis below we speculate the campaign with this malware started about a year ago, and it remained quite anonymous and undetected.

As per the main payload, it is named in the download server as avatar.php, after it is downloaded, it’s renamed to httpd. The machine is fingerprinted by various commands such as uname −a, then it starts unpacking itself.

Next, the httpd executable is copied from the running process into to /tmp directory, as illustrated in Figure 33 below. What’s interesting is that it finds the name of the process name that ran it, and saves itself under the /tmp directory with the same name. It also saves the pid under the /tmp/.apid. Lastly, httpd deletes itself.

Figure 33: httpd copies itself from memory

This technique is called “process masquerading” or “process replacement”. It is often done for the following reasons:

Defense Evasion: By deleting the original binary and copying itself to another location, the malware avoids detection from static file-based security measures that might be monitoring the original location. The /tmp directory is a common target because it is typically writable and frequently used for temporary files, making it less suspicious.
Obfuscation: Deleting the original binary and killing itself can make it harder for security analysts to trace back the activity to the original payload, thereby complicating forensic analysis.

Analysis of sh

The binary sh (MD5: 656e22c65bf7c04d87b5afbe52b8d800) is an exact copy of httpd. After sh is executed, it sleeps for 10 minutes. Next it collects information about the OS.

Next, sh drops nine binaries. Four are exact duplication of sh/httpd. A cryptominer and a rootkit (discussed below in ‘The Rootkit’ section). There are 3 lean binaries ldd, top and wizlmsh. The first 2 are user land rootkits, in some executions we also saw lsof and crontab. Wizlmsh is used to ensure the malware is running.

The malware opens a Unix socket to communicate with all the process it will run in the future. Via /tmp/.xdiag/int/.per.s, it writes logs, which will later be used by other dropped components as part of the attack.

The malware is also running various operations such as shutting down security controls, as seen in the example below:

Figure 34: Shutting down security controls

The binary sh is also copying itself from memory to various location, as illustrated below it saves itself as libpprocps.so and also as /root/.config/cron/perfcc, /usr/bin/perfcc, and /usr/lib/libfsnkdev.so.

Figure 35: sh copying itself from memory

This is a tactic used for persistence, stealth, and possibly for privilege escalation. Below we discuss the various path chosen:

The path /root/.config/cron/perfcc: This path is quite deceptive because it mimics a configuration directory under the root user, which might be overlooked by security scans assuming it’s a legitimate config file. The inclusion of cron in the path suggests an attempt to associate the malware with cron jobs.
The path /usr/bin/perfcc: The path /usr/bin is a standard directory for executable programs accessible to all users. Placing malware here could allow it to be executed like a normal system command, making detection harder. Naming the malware perfcc might be an attempt to masquerade as a legitimate system utility or command, reducing suspicion.
The binaries /usr/lib/libpprocps.so and /usr/lib/libfsnldev.so: These paths suggest the malware is impersonating shared libraries. /usr/lib is commonly used for storing shared libraries required by installed applications. The path libpprocps.so might be intended to appear related to the legitimate procps, a library and set of commands that includes utilities like ps, top, etc., which are used to display information about currently running processes.

The choice of these paths generally reflects a strategy to blend in with normal system operations, either by appearing as a utility or library that might regularly be executed or loaded by other processes.

Appendix 4: The main rootkit (libgcwrap.so)

The rootkit has several purposes. One of the main purposes is to hook various functions and modify their functionality. The rootkit itself is an ELF 64-bit LSB shared object (.so) file named libgcwrap.so. The rootkit is using LD_PRELOAD to load itself before other libraries.

As illustrated in Figure 36 below, the rootkit strings are encrypted with XOR and this function is iterating through an array, while performing XOR decryption on each element in the array.

Figure 36: XOR decrypt array

You can see in Figure 37 below the xor_decrypt function responsible to decrypt a string by iterating over each byte of the input string, doing XOR with the key 0xAC.

Figure 37: XOR decrypt

It does various interesting manipulations, including hooking to Libpam symbols. Specifically, to the function pam_authenticate, which is used by PAM to authenticate users. Hooking or overwriting this function could allow unauthorized actions during the authentication process, such as bypassing password checks, logging credentials, or modifying the behavior of authentication mechanisms.

In addition, the rootkit is also designed to hook Libpcap symbols, specifically to the function pcap_loop, which is widely used for capturing network traffic.

Below we discuss what the attacker is trying to do with these hookings:

Network Traffic Manipulation: By hooking pcap_loop, an attacker could alter the behavior of applications that rely on libpcap for capturing network traffic. This could include security monitoring tools, network analyzers, and other systems that perform packet analysis. Manipulating this function could lead to missed detections, altered traffic logs, or leakage of sensitive data.
Data Eavesdropping: The hooked function might be modified to stealthily copy certain data passing through the network to a location controlled by an attacker, effectively creating a data exfiltration pathway.
Persistence and Evasion: Placing malicious code in /tmp and hooking critical functions like pcap_loop can be part of a strategy to maintain persistence on a host with minimal detection. This setup allows an attacker to continue malicious activities even after primary payloads are detected and removed.

Appendix 5: User Land Rootkits ‘top’, ‘ldd’, ‘crontab’ and ‘lsof’

The malware Perfctl is dropping in the path /home/???/.local/bin/ 4 binaries. In our case top, ldd, lsof and crontab.

Below is the snippet executing top:

Figure 38: top execution

As seen in Figure 38 above, in the first row the script checks if the environment variable ABWTRX is set, if it exists this shell script won’t run. This is probably to check if the binary originally exists on the server or not.

Next the variables r and m are defined as top and perfctl accordingly. We assume that perfctl in this case is the object to hide which is the cryptominer.

Next the script iterates over the /.local/bin binaries to save them in the environment variable PATH. Now the script checks if the environment variable AAZHDE is set. Just a reminder. The initial script, when the server is first compromised, sets the environment variable AAZHDE. This is an inspection to verify that the “normal” course of the execution of this malware remains. Probably to evade sandbox execution.

If AAZHDE is set, the binary top is executed passing all the arguments that were passed during the script execution.

If AAZHDE isn’t set, the script executes various commands. Including, 2 trap commands. These trap commands execute deletion of the directory /tmp/smpr when the script exists or if it is interrupted by a user. It then recreates /tmp/smpr than it sets the environment variable AAZHDE as 1.

Next, the cryptominer is stopped, and the file /tmp/.apid is deleted. Finally, AAZHDE is unset and top is executed.

Lastly, the environment variable AAZHDE is unset and the binary top is executed.

Below is the snippet executing ldd:

Figure 39: ldd execution

This snippet is very similar to the one of top. As seen in Figure 37 above, in the first row the script checks if the environment variable ABWTRX is set, if it exists this shell script won’t run. This is probably to check if the cryptominer is running or not (we will get to that next), if the cryptominer is running this script exits, otherwise it runs and executes the cryptominer.

The binary top receives 2 parameters, as pointer to an executable and a pointer to the argv. It performs multiple steps including initialization, environment setup, cryptographic operations, data manipulation, and eventually executing another program. It also runs infinite loop to keep the process running in the background.

Figure 40: top binary

You can see two functions modify_lookup_table_with_offset and libc_xor_cipher, which are used to de-obfuscate various sections in memory. Next, there are several checks of the environment variables and errors.

Lastly, if all conditions are met, we see and execution of a binary (provided as argument during top execution).

Figure 41: top binary

Top is used for real-time monitoring of system performance and processes. Thus, if a developer encounters a slowdown in the system corresponding to cryptomining activity and asks to check the cpu of all running processes, the new tempered with top will not show the cryptominer’s cpu consumption.

Ldd is used to display the shared libraries (dynamic dependencies) required by an executable or a shared library. It shows which libraries an application depends on, as well as the paths to those libraries. The threat actor wants to hide malicious libraries and dependencies used by the malware, preventing detection during inspections.

Crontab is used to schedule and manage recurring tasks (cron jobs) to run at specified times on Linux/Unix systems.

lsof Lists open files and shows which processes are using them, including files, sockets, and network connections.

This makes perfect sense that the threat actor is trying to modify the results of these utilities as they may be used by developers or security engineers to evaluate the server and understand what is attacking the machine.