In this blog post, Aqua Nautilus researchers aim to shed light on a Linux malware that, over the past 3-4 years, has actively sought more than 20,000 types of misconfigurations in order to target and exploit Linux servers. If you have a Linux server connected to the internet, you could be at risk. In fact, given the scale, we strongly believe the attackers targeted millions worldwide with a potential number of victims of thousands, it appears that with this malware any Linux server could be at risk.
We discovered numerous incident reports in community forums, all describing indicators of compromise linked to this malware. The community has widely referred to it as the “perfctl malware,” and we have adopted this name.
This post will explore the malware’s architecture, components, defense evasion tactics, persistence mechanisms, and how we managed to detect it. Perfctl is particularly elusive and persistent, employing several sophisticated techniques, including:
- It utilizes rootkits to hide its presence.
- When a new user logs into the server, it immediately stops all “noisy” activities, lying dormant until the server is idle again.
- It utilizes Unix socket for internal communication and TOR for external communication.
- After execution, it deletes its binary and continues to run quietly in the background as a service.
- It copies itself from memory to various locations on the disk, using deceptive names.
- It opens a backdoor on the server and listens for TOR communications.
- It attempts to exploit the Polkit vulnerability (CVE-2021-4043) to escalate privileges.
In all the attacks observed, the malware was used to run a cryptominer, and in some cases, we also detected the execution of proxy-jacking software. During one of our sandbox tests, the threat actor utilized one of the malware’s backdoors to access the honeypot and started deploying some new utilities to better understand the nature of our server, trying to understand what exactly we are doing to its malware.
Elusive Malware Dominates Developer Forums
Our story begins with an attack we monitored on one of our honeypots. Typically, we check if anyone has already documented the attack, as this allows us to analyze it more thoroughly and compare our findings with those of other researchers. However, in this case, we found no report about the malware that had targeted our honeypot.
What we did find, though, were numerous references to a perfctl malware, this immediately drew our attention, as this was one of the names of our malware. These are in various languages across several developer communities and forums, we carefully reviewed these posts and found that the indicators of compromise mentioned in them are the same as the ones we’ve seen in our attack. Usually in some of these posts you can find replies with links to reports about the malware written by researchers. But in this case, however, none of these had links to such reports. Here are some of the posts we came across: Reddit, freelancer, Stack Overflow (Spanish), forobeta (Spanish), brainycp (Russian), natnetwork (Indonesian), Proxmox (Deutsch), Camel2243 (Chinese), svrforum (Korean), exabytes, virtualmin, serverfault and many others.
The name perfctl comes from the cryptominer process that drains the system’s resources, causing significant issues for many Linux developers. By combining “perf” (a Linux performance monitoring tool) with “ctl” (commonly used to indicate control in command-line tools), the malware authors crafted a name that appears legitimate. This makes it easier for users or administrators to overlook during initial investigations, as it blends in with typical system processes.
Towards the end of our research, we saw the first report covered by the researchers of Cado Security, but they only tell a very small part of the story of perfctl malware.
The Attack Flow
After exploiting a vulnerability (as in our case) or a misconfiguration, the main payload is downloaded from an HTTP server controlled by the attacker.
In our case, the main payload was named httpd, and it demonstrated multiple layers of execution, showcasing a deliberate design to ensure persistence and evade detection. Once executed, the main payload copies itself from memory to a new location in the ‘/tmp’ directory, runs the new binary from there, terminates the original process, and then deletes the initial binary to cover its tracks.
The main payload is now executed from the /tmp directory under a different name. Based on what we’ve seen the malware chose the name of the process that originally executed it, thus it looks less suspicious, if the system is examined.
In our case the malware was executed by sh, thus the name of the malware was changed from httpd to sh. At this point, it functions as both a dropper and a local command-and-control (C2) process. The malware contains an exploit to CVE-2021-4043, which it is trying to run in order to gain root privilege on the server.
The malware continues to copy itself from memory to half a dozen other locations, with names that appear as conventional system files. It also drops a rootkit and a few popular Linux utilities that were modified to serve as user land rootkits (i.e. ldd, lsof). A cryptominer is also dropped and in some executions, we also observed some proxy-jacking software transferred from a remote server and executed.
As part of its command-and-control operation, the malware opens a Unix socket, creates two directories under the /tmp directory, and stores data there that influences its operation. This data includes host events, locations of the copies of itself, process names, communication logs, tokens, and additional log information. Additionally, the malware uses environment variables to store data that further affects its execution and behavior.
All the binaries are packed, stripped, and encrypted, indicating significant efforts to bypass defense mechanisms and hinder reverse engineering attempts. The malware also uses advanced evasion techniques, such as suspending its activity when it detects a new user in the btmp or utmp files and terminating any competing malware to maintain control over the infected system.
Below is the complete attack flow:
Figure 1: The entire attack flow
As noted earlier, numerous files are written to disk or modified, primarily in the /tmp, /usr, and /root directories, as shown in the diagram below.
Figure 2: Files dropped or written to disk
In this blog and its appendices, we will explain the purpose of these files and the role each plays in the attack flow.
Perfctl Attack Highlights
The main binary httpd is a packed, stripped and obfuscated ELF (MD5: 656e22c65bf7c04d87b5afbe52b8d800). If you type the download url in the browser the integer 1 is printed to screen. If you try downloading the .php file without a specific user agent, you will receive a file with the integer 1. This response indicates that this file is completely innocent. But if you use the correct user agent it will drop the malware (size of ~9mb). This is a clever way to conceal the malware.
After it is downloaded and executed the malware copies itself from memory using another running process name, and it saves the process ID of that running process under /tmp/.apid.
Figure 3: httpd is copying itself from memory
Httpd then stops and deletes itself. This technique is called ‘process masquerading’ or ‘process replacement’ and it’s done for defense evasion and obfuscation. It can make security researchers life a bit harder to follow the malware execution flow.
The new Httpd binary is now saved in the /tmp directory under the name of the process that executed it sh in our case, but we’ve also seen other names when we used other processes to run it. The binary sh is also copying itself from memory to various locations, as it saves itself as libpprocps.so and also as /root/.config/cron/perfcc, /usr/bin/perfcc, and /usr/lib/libfsnkdev.so. In annex 3 – The main payload below, we discuss in detail about this and explain our hypothesis to why the threat actor chose these names. This shows of a thought in regard to persistence as the malware author creates a lot of locations to which the malware is copied.
Persistence
The attacker modifies the ~/.profile script, which sets up the environment during user login. This script is designed to execute the malware first, followed by the legitimate workload expected to run on the server. It checks if /root/.config/cron/perfcc is an executable file, and if so, it runs the malware.
Additionally, the script ensures that in Bash environments, the ~/.bashrc file is executed, applying user-specific configurations such as aliases and environment variables—likely to maintain normal server operations while the malware runs. Finally, the script suppresses mesg errors to avoid any visible warnings during execution.
The binary wizlmsh is dropped to /usr/bin (MD5: ba120e9c7f8896d9148ad37f02b0e3cb). It is a very small binary (12kb), that runs as a service in the background. Initially, it receives argc, and argv, and verify the execution of main payload (httpd) after it is written into /tmp either as sh or bash or any other name. It is responsible for the persistence of perfctl malware.
Figure 4: wizlmsh main function
Defense Evasion
The rootkit has several purposes. One of the main purposes is to hook various functions and modify their functionality. The rootkit itself is an ELF 64-bit LSB shared object (.so) file named libgcwrap.so (MD5: 835a9a6908409a67e51bce69f80dd58a). The rootkit is using LD_PRELOAD to load itself before other libraries.
Figure 5: The revised LD_Preload content
It does various interesting manipulations, including hooking to Libpam symbols. Specifically, to the function pam_authenticate, which is used by PAM to authenticate users. Hooking or overwriting this function could allow unauthorized actions during the authentication process, such as bypassing password checks, logging credentials, or modifying the behavior of authentication mechanisms.
Figure 6: Functions the rootkit hooks
In addition, the rootkit is also designed to hook Libpcap symbols, specifically to the function pcap_loop, which is widely used for capturing network traffic. This is used to prevent recording of the malware traffic.
The threat actors are also using a few user land rootkits. They drop a few legitimate utilities such as ldd. These utilities were modified to hide specific attack elements. So, if the rigged crontab is used, for instance, it won’t show cron jobs created during the attack.
In the first step the malware replaces the /etc/profile so the path will be set on /bin/.local/bin:$PATH. In this path the threat actor is bypassing the directory where the utilities are called from. We’ve seen in some malware runs 2 binaries and in other 4 binaries, depending on which utilities exist originally on the server.
In our attacks the malware dropped crontab, lsof, ldd and top. These tweaked binaries will hide malicious activities, in case someone is using them.
Figure 7: The new content inserted by the threat actor to ‘/etc/profile’
In appendix 5 – User land rootkits we explain in detail why we think these utilities were chosen by the threat actor.
Main Impact
The main impact of the attack is resource hijacking. In all cases we observed a monero cryptominer (XMRIG) executed and exhausting the server’s CPU resources. The cryptominer is also packed and encrypted. Once unpacked and decrypted it communicates with cryptomining pools.
As reflected in Figure 8 below, the cryptomining pools are accessed via TOR.
Figure 8: Cryptomining traffic
Moreover, in some of the attacks we’ve seen proxy-jacking via various vendors. We’ve seen the communication with the following domains: bitping.com, earn.fm, speedshare.app, and repocket.com.
The domain repocket.com, for instance, is associated with the Repocket platform, which is a service that allows users to earn money by sharing their unused internet bandwidth.
In addition, we can observe the usage of the bitping daemon usage, which provide similar bandwidth payment services.
Figure 9: Logging in to bitping
TOR communication
The binary sh is also initiating communication via Tor with few servers (i.e. 80.67.172.162, 176.10.107.180, 78.47.18.110, 95.217.109.36, 145.239.41.102).
While the communication is encrypted, you can observe the TOR log left on our honeypot.
Figure 10: TOR sessions log
Additional Threat Intelligence
We recorded several dozen attacks of perfctl. We saw 3 download servers involved in these attacks (46.101.139.173, 104.183.100.189 and 198.211.126.180).
The first two IP addresses seem to be linked to vulnerable servers that were previously hacked by the threat actor and the third one could be owned by the threat actor. All 3 IP addresses store and hide artifacts used in this campaign.
In most of the attacks we see that the binaries were dropped from IP address 46.101.139.173. An inspection of this IP address showed that this is a compromised webserver.
Figure 11: Compromised website serves as download server
Iterating over this download server, we see a compromised site on a server in Germany.
We noticed some artifacts, well-hidden between the site’s scripts. We see 3 main payloads. One is avatar.php, which was used as part of the attack on our honeypot. When using the browser to reach to the webpage with avatar.php or downloading it without a specific user agent leads to 1 being displayed of screen or a .php file with the digit 1.
In addition, there is another file named aoip, which was uploaded 2 months later and two others dark.css and csdark.css which were uploaded later.
Figure 12: Files hosted on the webserver
Figure 13: Files hosted on the webserver
Figure 14: Files hosted on the webserver
The binary aoip is a replication of the main payload (sh/httpd).
Csdark.css and dark.css weren’t analyzed during this research but look very interesting.
On IP address 198.211.126.180 we found just the file checklist.php which is the main payload (sh/httpd).
Figure 15: Compromised website serves as download server
On IP address 104.183.100.189 we found another innocent compromised website.
Figure 16: Compromised website serves as download server
It looks like this website stores this XML file which when decoded (base64) is actually the rconf script.
Figure 17: malicious XML
From what we see on these websites, there are few artifacts used to execute the exploitation of misconfigured and vulnerable (in our recorded attacks) Linux servers. We identified a very long list of almost 20K directory traversal fuzzing list, seeking for mistakenly exposed configuration files and secrets. There are also a couple of follow-up files (such as the XML) the attacker can run to exploit the misconfiguration. In the table below you can see the analysis of the paths, which shows that perfctl is mainly looking to exploit misconfigurations.
Continuation of:
Source: