Got 10 VestaCP servers exploited
Re: Got 10 VestaCP servers exploited
while this is ofc some good advice and true that it is dangerous to have them open,dpeca wrote: ↑Tue Apr 17, 2018 10:52 pmHere is my list of disabled functions in php.ini:My server never got rooted from PHP level with this.Code: Select all
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,exec,system,passthru,shell_exec,proc_open,popen
i got hacked with disabled exec and shell_exec.
Re: Got 10 VestaCP servers exploited
proc_open & popen is not in my global disabled list, from what u have posted.
but thats for a reason. i cannot disable it for everyone on my server.
edit: forget the rest i posted after this, since it didnt work anyway
Re: Got 10 VestaCP servers exploited
Very interesting approach! I'm posting below my own findings. Maybe there is a correlation after all.
NON-Compromised Server(s)
Code: Select all
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,exec,passthru,shell_exec,system,proc_open,popen,curl_multi_exec,show_source,
Code: Select all
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,
In the non-compromised server(s) the following functions are disabled:
Code: Select all
exec,passthru,shell_exec,system,proc_open,popen,curl_multi_exec,show_source
- I have at least one not-compromised server where these functions are actually enabled.
- curl_exec is needed by Joomla and others
- shell_exec,parse_ini_file needed by Matomo (Piwik)
-
- Support team
- Posts: 1096
- Joined: Sat Sep 06, 2014 9:58 pm
- Contact:
- Os: Debian 8x
- Web: apache + nginx
Re: Got 10 VestaCP servers exploited
Luckily,
I had only one server hacked with gcc.sh shit, I was able to clean the infection without a format. This server is only hosting one CakePHP site... completely locked down on security and always had Vesta port obfuscated, right from the time of install, yet it still it got hacked. My initial and final thought is again PHP API which was exploited. The hacker sniffed IP Ranges, found Vesta's API listening on a TCP port and when found, explited the API Call. Though I'm not a coder myself to interpret the vector of the attack correctly
What you say Dpeca is good for 75% of VestaCP users, I think as a good security practice we can have PHP shell access forbidden by default. This way, you have a smaller attack surface open by default... and if any of your apps doesn't work without them, you know it very well on which server you have them enabled. So I vote for the though to have them disabled after default VestaCP install.
-
- Posts: 1
- Joined: Wed Jun 03, 2015 7:22 am
Re: Got 10 VestaCP servers exploited
In my case, I was wondering why the virus was stored in this folder:
/tmp/systemd-private-72defd09721c49848b659469def3d414-httpd.service-4ZhWF5/tmp/
Following this explanation, http://0pointer.de/blog/projects/security.html, so it looks like it was uploaded using Apache service (httpd.service) and not nginx + php-fpm (Vestacp service).
My Vestacp installation parameters were:
========
bash vst-install.sh --nginx no --apache yes --phpfpm no --named yes --remi yes --vsftpd yes --proftpd no --iptables yes --fail2ban no --quota yes --exim yes --dovecot yes --spamassassin yes --clamav yes --softaculous no --mysql yes --postgresql no --hostname xxxx.xxxxxxxx.com --email [email protected]
========
This is the warning email that I received from my firewall (CSF Firewall):
========
Subject: lfd on xxxx.xxxxxx.com: Suspicious File Alert
Time: Wed Apr 4 06:30:07 2018 -0400
File: /tmp/systemd-private-72defd09721c49848b659469def3d414-httpd.service-4ZhWF5/tmp/update
Reason: Linux Binary
Owner: admin:admin (1002:1002)
Action: No action taken
========
Also I was wondering why the user and group were "admin:admin". And I think this only happens when:
1) The malicious code is executed via VestaCP service. I doubt it. Furthermore, in my case I had vestacp port service disabled in the firewall by default (8083). Only my fixed IP was allowed to reach it.
2) The malicious code is executed via hostname/IP service, like http://xxx.xxx.xxx.xxx
So my conclusion is that one of the possibles vector attacks was using http://xxx.xxx.xxx.xxx and NOT vestacp service.
The pity things are:
1) I have not records in apache logs of it. They are missing or deleted or never created. I see only a /webmail/ HEAD access (from a japanese IP) 3 seconds before that "update" file was created in that "tmp" folder:
========
119.82.29.17 - - [04/Apr/2018:06:25:38 -0400] "HEAD /webmail/ HTTP/1.0" 200 - "http://www.google.com" "Googlebot/2.1 (+http://www.google.com/bot.html)
========
2) I am not sure if this is vestacp API related or not. Not sure if there is a way to access to API calls from Apache (for websites) instead ngnix from port 8083 (for vesta).
I hope this information helps to find the vector attack.
In order to remove it, I followed these steps:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
chmod 0000 /lib/libudev.so
rm -f /lib/libudev.so
chattr +i /lib/
chmod 0000 /etc/cron.hourly/gcc.sh
rm -f /etc/cron.hourly/gcc.sh
========
Then I have looked for strange names in /usr/bin (because I had 4 virus entries there) and then I have removed them in this way:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
find /etc -name '*uwdvmfnywd*'| xargs rm -f
find /etc -name '*eijaxxglfs*'| xargs rm -f
find /etc -name '*ffoljgwzyq*'| xargs rm -f
find /etc -name '*update*' | grep "rc.d" | xargs rm -f
rm -f /usr/bin/uwdvmfnywd
rm -f /usr/bin/eijaxxglfs
rm -f /usr/bin/ffoljgwzyq
rm -f /usr/bin/update
========
Then I have killed the virus process (if any active at that moment):
======== DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
pkill uwdvmfnywd
========
Take in consideration that while you are doing this, the virus is creating new entries in /usr/bin and everything, but it won't be able to run again because the "chattr +i /lib/" previous command you issued. So be calm, you should remove them again later. Now continue reading...
Then you should edit the crontab:
======== DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
vi /etc/crontab
========
and remove or comment the gcc.sh entry.
Then, in my case, I had to kill "sleep" pid process that was causing to reproduce the virus every 5 seconds.
So I did something like:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
ps -auxf
=======
and I identified the "sleep" process at the end of the list (could be upper, but not too much) and the I killed it:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
kill -9 <pid>
=======
Once killed I have repeated once this process from the beginning skipping the "libudev" part because it was not created again.
When you are done remember to change attributes to /lib/ folder:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
chattr -i /lib/
=======
And check again the /etc/cron-hourly folder just in case there are still remaining virus running or not.
Also I had to reload my systemd daemon because the virus was creating services entries there that automatically will be removed when reloading them in this way:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
systemctl daemon-reload
======
I had not to reboot the machine using this procedure and I am not noticing further problems since them (two days ago).
Hope it helps.
/tmp/systemd-private-72defd09721c49848b659469def3d414-httpd.service-4ZhWF5/tmp/
Following this explanation, http://0pointer.de/blog/projects/security.html, so it looks like it was uploaded using Apache service (httpd.service) and not nginx + php-fpm (Vestacp service).
My Vestacp installation parameters were:
========
bash vst-install.sh --nginx no --apache yes --phpfpm no --named yes --remi yes --vsftpd yes --proftpd no --iptables yes --fail2ban no --quota yes --exim yes --dovecot yes --spamassassin yes --clamav yes --softaculous no --mysql yes --postgresql no --hostname xxxx.xxxxxxxx.com --email [email protected]
========
This is the warning email that I received from my firewall (CSF Firewall):
========
Subject: lfd on xxxx.xxxxxx.com: Suspicious File Alert
Time: Wed Apr 4 06:30:07 2018 -0400
File: /tmp/systemd-private-72defd09721c49848b659469def3d414-httpd.service-4ZhWF5/tmp/update
Reason: Linux Binary
Owner: admin:admin (1002:1002)
Action: No action taken
========
Also I was wondering why the user and group were "admin:admin". And I think this only happens when:
1) The malicious code is executed via VestaCP service. I doubt it. Furthermore, in my case I had vestacp port service disabled in the firewall by default (8083). Only my fixed IP was allowed to reach it.
2) The malicious code is executed via hostname/IP service, like http://xxx.xxx.xxx.xxx
So my conclusion is that one of the possibles vector attacks was using http://xxx.xxx.xxx.xxx and NOT vestacp service.
The pity things are:
1) I have not records in apache logs of it. They are missing or deleted or never created. I see only a /webmail/ HEAD access (from a japanese IP) 3 seconds before that "update" file was created in that "tmp" folder:
========
119.82.29.17 - - [04/Apr/2018:06:25:38 -0400] "HEAD /webmail/ HTTP/1.0" 200 - "http://www.google.com" "Googlebot/2.1 (+http://www.google.com/bot.html)
========
2) I am not sure if this is vestacp API related or not. Not sure if there is a way to access to API calls from Apache (for websites) instead ngnix from port 8083 (for vesta).
I hope this information helps to find the vector attack.
In order to remove it, I followed these steps:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
chmod 0000 /lib/libudev.so
rm -f /lib/libudev.so
chattr +i /lib/
chmod 0000 /etc/cron.hourly/gcc.sh
rm -f /etc/cron.hourly/gcc.sh
========
Then I have looked for strange names in /usr/bin (because I had 4 virus entries there) and then I have removed them in this way:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
find /etc -name '*uwdvmfnywd*'| xargs rm -f
find /etc -name '*eijaxxglfs*'| xargs rm -f
find /etc -name '*ffoljgwzyq*'| xargs rm -f
find /etc -name '*update*' | grep "rc.d" | xargs rm -f
rm -f /usr/bin/uwdvmfnywd
rm -f /usr/bin/eijaxxglfs
rm -f /usr/bin/ffoljgwzyq
rm -f /usr/bin/update
========
Then I have killed the virus process (if any active at that moment):
======== DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
pkill uwdvmfnywd
========
Take in consideration that while you are doing this, the virus is creating new entries in /usr/bin and everything, but it won't be able to run again because the "chattr +i /lib/" previous command you issued. So be calm, you should remove them again later. Now continue reading...
Then you should edit the crontab:
======== DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
vi /etc/crontab
========
and remove or comment the gcc.sh entry.
Then, in my case, I had to kill "sleep" pid process that was causing to reproduce the virus every 5 seconds.
So I did something like:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
ps -auxf
=======
and I identified the "sleep" process at the end of the list (could be upper, but not too much) and the I killed it:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
kill -9 <pid>
=======
Once killed I have repeated once this process from the beginning skipping the "libudev" part because it was not created again.
When you are done remember to change attributes to /lib/ folder:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
chattr -i /lib/
=======
And check again the /etc/cron-hourly folder just in case there are still remaining virus running or not.
Also I had to reload my systemd daemon because the virus was creating services entries there that automatically will be removed when reloading them in this way:
======= DO NOT PASTE/COPY. INVESTIGATE YOUR CASE FIRST
systemctl daemon-reload
======
I had not to reboot the machine using this procedure and I am not noticing further problems since them (two days ago).
Hope it helps.
Re: Got 10 VestaCP servers exploited
I second this, definitely related somehow!masterguru wrote: ↑Sat Apr 21, 2018 9:31 amThe pity things are:
1) I have not records in apache logs of it. They are missing or deleted or never created. I see only a /webmail/ HEAD access (from a japanese IP) 3 seconds before that "update" file was created in that "tmp" folder:
========
119.82.29.17 - - [04/Apr/2018:06:25:38 -0400] "HEAD /webmail/ HTTP/1.0" 200 - "http://www.google.com" "Googlebot/2.1 (+http://www.google.com/bot.html)
========
for this:
keep in mind that an initial attack could have tinkered with those firewall restrictions and after that any api-call could have processed the files placed in /tmp and gained elevated access...1) The malicious code is executed via VestaCP service. I doubt it. Furthermore, in my case I had vestacp port service disabled in the firewall by default (8083). Only my fixed IP was allowed to reach it.
Re: Got 10 VestaCP servers exploited
Yep, another exploited VCP server from my client geared by his previous sysadmin, I just move away the website into our cpanel server...
Re: Got 10 VestaCP servers exploited
Some basic tips for everyone: (for details you can google for the keywords and your OS)
-) Keep your distro packages up-to-date with mail-warnings on available updates
-) Harden your ssh with pubkeys and disable root-login (setup sudo)
-) Setup mail warning on ssh login
-) Close vestacp admin port to specific ip adresses (your ip)
-) Put basic htaccess @ phpmyadmin and roundcoube (give details to clients)
-) Disable functions in php (if not needed): exec,shell_exec,passthru,system,proc_open,popen,show_source
-) Setup clamav, chkrootkit, rkhunter, aide, csf and/or similar + regular updates and custom scan routines via cron (antivirus, firewall and intrusion detection)
With this steps you should be way ahead to when you didnt have all of this done.
Ofc, theres alot more you can do, but this things from above aren't hard to accomplish and add a good layer of security!
-) Keep your distro packages up-to-date with mail-warnings on available updates
-) Harden your ssh with pubkeys and disable root-login (setup sudo)
-) Setup mail warning on ssh login
-) Close vestacp admin port to specific ip adresses (your ip)
-) Put basic htaccess @ phpmyadmin and roundcoube (give details to clients)
-) Disable functions in php (if not needed): exec,shell_exec,passthru,system,proc_open,popen,show_source
-) Setup clamav, chkrootkit, rkhunter, aide, csf and/or similar + regular updates and custom scan routines via cron (antivirus, firewall and intrusion detection)
With this steps you should be way ahead to when you didnt have all of this done.
Ofc, theres alot more you can do, but this things from above aren't hard to accomplish and add a good layer of security!
-
- Posts: 73
- Joined: Sun Dec 03, 2017 6:30 pm
Re: Got 10 VestaCP servers exploited
Disable functions like dpeca suggested. And implement mod security