Page 53 of 55

Re: Got 10 VestaCP servers exploited

Posted: Wed Apr 18, 2018 7:10 am
by kobo1d
dpeca wrote:
Tue Apr 17, 2018 10:52 pm
Here is my list of disabled functions in php.ini:

Code: Select all

disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,exec,system,passthru,shell_exec,proc_open,popen
My server never got rooted from PHP level with this.
while this is ofc some good advice and true that it is dangerous to have them open,
i got hacked with disabled exec and shell_exec.

Re: Got 10 VestaCP servers exploited

Posted: Wed Apr 18, 2018 7:28 am
by dpeca
kobo1d wrote:
Wed Apr 18, 2018 7:10 am
while this is ofc some good advice and true that it is dangerous to have them open,
i got hacked with disabled exec and shell_exec.
system, passthru, proc_open and popen was enabled?

Re: Got 10 VestaCP servers exploited

Posted: Wed Apr 18, 2018 7:42 am
by kobo1d
dpeca wrote:
Wed Apr 18, 2018 7:28 am
kobo1d wrote:
Wed Apr 18, 2018 7:10 am
while this is ofc some good advice and true that it is dangerous to have them open,
i got hacked with disabled exec and shell_exec.
system, passthru, proc_open and popen was enabled?
proc_open & popen is not in my global disabled list, from what u have posted.
but thats for a reason. i cannot disable it for everyone on my server.

edit: forget the rest i posted after this, since it didnt work anyway

Re: Got 10 VestaCP servers exploited

Posted: Wed Apr 18, 2018 1:02 pm
by Felix
dpeca wrote:
Tue Apr 17, 2018 6:37 pm
I, personally, have one question for all administrators whose server got hacked.

Did you disabled dangerous PHP functions (like shell_exec(), system() and exec()) with "disable_functions" in php.ini ?
Very interesting approach! I'm posting below my own findings. Maybe there is a correlation after all.

NON-Compromised Server(s)

Code: Select all

disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,exec,passthru,shell_exec,system,proc_open,popen,curl_multi_exec,show_source,
Compromised Server(s)

Code: Select all

disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,
You might want to compare them with online diff tools like https://text-compare.com/ or http://rextester.com/diff to easily spot the differences.

In the non-compromised server(s) the following functions are disabled:

Code: Select all

exec,passthru,shell_exec,system,proc_open,popen,curl_multi_exec,show_source
Some notes
  • I have at least one not-compromised server where these functions are actually enabled.
  • curl_exec is needed by Joomla and others
  • shell_exec,parse_ini_file needed by Matomo (Piwik)

Re: Got 10 VestaCP servers exploited

Posted: Wed Apr 18, 2018 7:15 pm
by mehargags
dpeca wrote:
Wed Apr 18, 2018 7:28 am
kobo1d wrote:
Wed Apr 18, 2018 7:10 am
while this is ofc some good advice and true that it is dangerous to have them open,
i got hacked with disabled exec and shell_exec.
system, passthru, proc_open and popen was enabled?
Luckily,
I had only one server hacked with gcc.sh shit, I was able to clean the infection without a format. This server is only hosting one CakePHP site... completely locked down on security and always had Vesta port obfuscated, right from the time of install, yet it still it got hacked. My initial and final thought is again PHP API which was exploited. The hacker sniffed IP Ranges, found Vesta's API listening on a TCP port and when found, explited the API Call. Though I'm not a coder myself to interpret the vector of the attack correctly

What you say Dpeca is good for 75% of VestaCP users, I think as a good security practice we can have PHP shell access forbidden by default. This way, you have a smaller attack surface open by default... and if any of your apps doesn't work without them, you know it very well on which server you have them enabled. So I vote for the though to have them disabled after default VestaCP install.

Re: Got 10 VestaCP servers exploited

Posted: Sat Apr 21, 2018 9:31 am
by masterguru
In my case, I was wondering why the virus was stored in this folder:

/tmp/systemd-private-72defd09721c49848b659469def3d414-httpd.service-4ZhWF5/tmp/

Following this explanation, http://0pointer.de/blog/projects/security.html, so it looks like it was uploaded using Apache service (httpd.service) and not nginx + php-fpm (Vestacp service).

My Vestacp installation parameters were:
========
bash vst-install.sh --nginx no --apache yes --phpfpm no --named yes --remi yes --vsftpd yes --proftpd no --iptables yes --fail2ban no --quota yes --exim yes --dovecot yes --spamassassin yes --clamav yes --softaculous no --mysql yes --postgresql no --hostname xxxx.xxxxxxxx.com --email xxxx@xxxxxxxxxxxxxx.com
========

This is the warning email that I received from my firewall (CSF Firewall):
========
Subject: lfd on xxxx.xxxxxx.com: Suspicious File Alert

Time: Wed Apr 4 06:30:07 2018 -0400
File: /tmp/systemd-private-72defd09721c49848b659469def3d414-httpd.service-4ZhWF5/tmp/update
Reason: Linux Binary
Owner: admin:admin (1002:1002)
Action: No action taken
========

Also I was wondering why the user and group were "admin:admin". And I think this only happens when:

1) The malicious code is executed via VestaCP service. I doubt it. Furthermore, in my case I had vestacp port service disabled in the firewall by default (8083). Only my fixed IP was allowed to reach it.

2) The malicious code is executed via hostname/IP service, like http://xxx.xxx.xxx.xxx

So my conclusion is that one of the possibles vector attacks was using http://xxx.xxx.xxx.xxx and NOT vestacp service.

The pity things are:
1) I have not records in apache logs of it. They are missing or deleted or never created. I see only a /webmail/ HEAD access (from a japanese IP) 3 seconds before that "update" file was created in that "tmp" folder:
========
119.82.29.17 - - [04/Apr/2018:06:25:38 -0400] "HEAD /webmail/ HTTP/1.0" 200 - "http://www.google.com" "Googlebot/2.1 (+http://www.google.com/bot.html)
========

2) I am not sure if this is vestacp API related or not. Not sure if there is a way to access to API calls from Apache (for websites) instead ngnix from port 8083 (for vesta).

I hope this information helps to find the vector attack.

In order to remove it, I followed these steps:

======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
chmod 0000 /lib/libudev.so
rm -f /lib/libudev.so
chattr +i /lib/
chmod 0000 /etc/cron.hourly/gcc.sh
rm -f /etc/cron.hourly/gcc.sh
========

Then I have looked for strange names in /usr/bin (because I had 4 virus entries there) and then I have removed them in this way:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
find /etc -name '*uwdvmfnywd*'| xargs rm -f
find /etc -name '*eijaxxglfs*'| xargs rm -f
find /etc -name '*ffoljgwzyq*'| xargs rm -f
find /etc -name '*update*' | grep "rc.d" | xargs rm -f

rm -f /usr/bin/uwdvmfnywd
rm -f /usr/bin/eijaxxglfs
rm -f /usr/bin/ffoljgwzyq
rm -f /usr/bin/update
========

Then I have killed the virus process (if any active at that moment):
======== DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
pkill uwdvmfnywd
========

Take in consideration that while you are doing this, the virus is creating new entries in /usr/bin and everything, but it won't be able to run again because the "chattr +i /lib/" previous command you issued. So be calm, you should remove them again later. Now continue reading...

Then you should edit the crontab:
======== DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
vi /etc/crontab
========
and remove or comment the gcc.sh entry.

Then, in my case, I had to kill "sleep" pid process that was causing to reproduce the virus every 5 seconds.

So I did something like:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
ps -auxf
=======

and I identified the "sleep" process at the end of the list (could be upper, but not too much) and the I killed it:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
kill -9 <pid>
=======

Once killed I have repeated once this process from the beginning skipping the "libudev" part because it was not created again.

When you are done remember to change attributes to /lib/ folder:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
chattr -i /lib/
=======

And check again the /etc/cron-hourly folder just in case there are still remaining virus running or not.

Also I had to reload my systemd daemon because the virus was creating services entries there that automatically will be removed when reloading them in this way:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
systemctl daemon-reload
======

I had not to reboot the machine using this procedure and I am not noticing further problems since them (two days ago).

Hope it helps.

Re: Got 10 VestaCP servers exploited

Posted: Sat Apr 21, 2018 11:18 am
by Falzo
masterguru wrote:
Sat Apr 21, 2018 9:31 am
The pity things are:
1) I have not records in apache logs of it. They are missing or deleted or never created. I see only a /webmail/ HEAD access (from a japanese IP) 3 seconds before that "update" file was created in that "tmp" folder:
========
119.82.29.17 - - [04/Apr/2018:06:25:38 -0400] "HEAD /webmail/ HTTP/1.0" 200 - "http://www.google.com" "Googlebot/2.1 (+http://www.google.com/bot.html)
========
I second this, definitely related somehow!

for this:
1) The malicious code is executed via VestaCP service. I doubt it. Furthermore, in my case I had vestacp port service disabled in the firewall by default (8083). Only my fixed IP was allowed to reach it.
keep in mind that an initial attack could have tinkered with those firewall restrictions and after that any api-call could have processed the files placed in /tmp and gained elevated access...

Re: Got 10 VestaCP servers exploited

Posted: Sat Apr 21, 2018 2:59 pm
by DevBox
Yep, another exploited VCP server from my client geared by his previous sysadmin, I just move away the website into our cpanel server...

Re: Got 10 VestaCP servers exploited

Posted: Sun Apr 22, 2018 6:52 am
by kobo1d
Some basic tips for everyone: (for details you can google for the keywords and your OS)

-) Keep your distro packages up-to-date with mail-warnings on available updates
-) Harden your ssh with pubkeys and disable root-login (setup sudo)
-) Setup mail warning on ssh login
-) Close vestacp admin port to specific ip adresses (your ip)
-) Put basic htaccess @ phpmyadmin and roundcoube (give details to clients)
-) Disable functions in php (if not needed): exec,shell_exec,passthru,system,proc_open,popen,show_source
-) Setup clamav, chkrootkit, rkhunter, aide, csf and/or similar + regular updates and custom scan routines via cron (antivirus, firewall and intrusion detection)

With this steps you should be way ahead to when you didnt have all of this done.
Ofc, theres alot more you can do, but this things from above aren't hard to accomplish and add a good layer of security!

Re: Got 10 VestaCP servers exploited

Posted: Sun Apr 22, 2018 9:59 am
by lukapaunovic
Disable functions like dpeca suggested. And implement mod security