Re: Got 10 VestaCP servers exploited
Posted: Mon Apr 09, 2018 10:16 am
Will be checked
Community Forum
https://forum.vestacp.com/
Someone said in this post that was hacked even not having vesta service in default port 8083. It makes sense that may be its done within the root account.wildwolf wrote: ↑Mon Apr 09, 2018 9:59 amJudging by audit.log's I have from several infected servers, it seems to me that it is not VestaCP that was compromised but its repository / repositories.
For example,
(bash - my login).Code: Select all
# ausearch -m USER_CMD -i | grep -v -- '----' | awk '{print $10}' | sort -u cmd=-bash cmd=/usr/local/vesta/bin/v-add-firewall-rule cmd=/usr/local/vesta/bin/v-add-letsencrypt-domain cmd=/usr/local/vesta/bin/v-add-user cmd=/usr/local/vesta/bin/v-add-web-domain cmd=/usr/local/vesta/bin/v-backup-users cmd=/usr/local/vesta/bin/v-change-user-language cmd=/usr/local/vesta/bin/v-change-web-domain-tpl cmd=/usr/local/vesta/bin/v-check-user-password cmd=/usr/local/vesta/bin/v-delete-domain cmd=/usr/local/vesta/bin/v-delete-mail-domain cmd=/usr/local/vesta/bin/v-list-databases cmd=/usr/local/vesta/bin/v-list-firewall cmd=/usr/local/vesta/bin/v-list-mail-account cmd=/usr/local/vesta/bin/v-list-mail-accounts cmd=/usr/local/vesta/bin/v-list-mail-domain cmd=/usr/local/vesta/bin/v-list-mail-domains cmd=/usr/local/vesta/bin/v-list-sys-config cmd=/usr/local/vesta/bin/v-list-sys-ips cmd=/usr/local/vesta/bin/v-list-sys-languages cmd=/usr/local/vesta/bin/v-list-sys-users cmd=/usr/local/vesta/bin/v-list-user cmd=/usr/local/vesta/bin/v-list-user-backups cmd=/usr/local/vesta/bin/v-list-user-favourites cmd=/usr/local/vesta/bin/v-list-user-ips cmd=/usr/local/vesta/bin/v-list-user-notifications cmd=/usr/local/vesta/bin/v-list-user-packages cmd=/usr/local/vesta/bin/v-list-users cmd=/usr/local/vesta/bin/v-list-users-stats cmd=/usr/local/vesta/bin/v-list-web-domain cmd=/usr/local/vesta/bin/v-list-web-domains cmd=/usr/local/vesta/bin/v-list-web-domain-ssl cmd=/usr/local/vesta/bin/v-list-web-stats cmd=/usr/local/vesta/bin/v-list-web-templates cmd=/usr/local/vesta/bin/v-list-web-templates-backend cmd=/usr/local/vesta/bin/v-restart-web cmd=/usr/local/vesta/bin/v-update-letsencrypt-ssl cmd=/usr/local/vesta/bin/v-update-sys-queue cmd=/usr/local/vesta/bin/v-update-sys-rrd cmd=/usr/local/vesta/bin/v-update-sys-vesta-all cmd=/usr/local/vesta/bin/v-update-user-stats
I checked arguments to every command, they were all OK, no traces of injection attempts etc.
This makes me think that infection took place from root account (by default, auditd does not log commands run by root): if VestaCP were vulnerable, I think I would see something suspicious either in audit.log or in secure / auth.log. I didn't, therefore either the infection took place long ago (I have logs back to Mar 13) -- but in this case someone should have noticed something strange -- libudev.so is detected even by ClamAV, or the infection was performed with root privileges. If so, need to find what was in common between the affected servers. So far, I know that they all run VestaCP (I heard of infected Debian servers). For infection to happen via root account, it either needs to be downloaded and run by the user itself (does anybody verify the checksum of the Vesta installer?), or come from the repository. This can explain the fact that there are no new infections — old (compromised?) files were overwritten with new ones.
I would recommend the VestaCP team to take a look at their infrastructure and see if there are any signs of compromise / unauthorized access.
Thanks for all the hard work. It seems my Vesta was already updated. I am on CentOS DigitalOcean and when I try doing yum update, there's no Vesta update. When I try using Vesta GUI, there is no update (see image https://puu.sh/zZsbV/3509057e9c.png).skid wrote: ↑Sun Apr 08, 2018 7:05 amHere is what we know so far:
1. The first wave happened on April 4. Servers were infected with /etc/cron.hourly/gcc.sh
2. It was an automated hack
3. CentOS, Debian, Ubuntu all distros are affected it's platform independent
4. We didn't find any traces in vesta and system logs yet
5. On April 7 infected servers started to DDoS remote hosts using /usr/lib/libudev.so.
What you can do:
The best way to stay safe is to temporary disable vesta web serviceCode: Select all
service vesta stop
or limit access to port 8083 using firewallCode: Select all
systemctl disable vesta
What we are doing:
Few users provided us with root access to their servers. We are investigating what happened. We also launched a couple honeypots in order to get full picture of the hack.
it happened to me twice when I made install at Bandwagon that ioncube was not properly installed. Seems sort of unable to locate files or sth. But the whole installation went to the final stage. Guess it might be someone hacked in?fedekrum wrote: ↑Mon Apr 09, 2018 10:14 amI have just tried to make a new vesta server on Digital Ocean, Ubuntu 16 and got these errors during install.
Hit:1 http://apt.vestacp.com/xenial xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:3 https://repos.sonar.digitalocean.com/apt main InRelease
Hit:4 http://nginx.org/packages/mainline/ubuntu xenial InRelease
Hit:5 http://nyc2.mirrors.digitalocean.com/ubuntu xenial InRelease
Hit:6 http://nyc2.mirrors.digitalocean.com/ubuntu xenial-updates InRelease
Hit:7 http://nyc2.mirrors.digitalocean.com/ubuntu xenial-backports InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package vesta-php
E: Unable to locate package vesta-ioncube
E: Unable to locate package vesta-softaculous
Error: apt-get install failed
Do you think it has to do with this hack or the patch released?
fedekrum wrote: ↑Mon Apr 09, 2018 10:40 amSomeone said in this post that was hacked even not having vesta service in default port 8083. It makes sense that may be its done within the root account.wildwolf wrote: ↑Mon Apr 09, 2018 9:59 amJudging by audit.log's I have from several infected servers, it seems to me that it is not VestaCP that was compromised but its repository / repositories.
For example,
(bash - my login).Code: Select all
# ausearch -m USER_CMD -i | grep -v -- '----' | awk '{print $10}' | sort -u cmd=-bash cmd=/usr/local/vesta/bin/v-add-firewall-rule cmd=/usr/local/vesta/bin/v-add-letsencrypt-domain cmd=/usr/local/vesta/bin/v-add-user cmd=/usr/local/vesta/bin/v-add-web-domain cmd=/usr/local/vesta/bin/v-backup-users cmd=/usr/local/vesta/bin/v-change-user-language cmd=/usr/local/vesta/bin/v-change-web-domain-tpl cmd=/usr/local/vesta/bin/v-check-user-password cmd=/usr/local/vesta/bin/v-delete-domain cmd=/usr/local/vesta/bin/v-delete-mail-domain cmd=/usr/local/vesta/bin/v-list-databases cmd=/usr/local/vesta/bin/v-list-firewall cmd=/usr/local/vesta/bin/v-list-mail-account cmd=/usr/local/vesta/bin/v-list-mail-accounts cmd=/usr/local/vesta/bin/v-list-mail-domain cmd=/usr/local/vesta/bin/v-list-mail-domains cmd=/usr/local/vesta/bin/v-list-sys-config cmd=/usr/local/vesta/bin/v-list-sys-ips cmd=/usr/local/vesta/bin/v-list-sys-languages cmd=/usr/local/vesta/bin/v-list-sys-users cmd=/usr/local/vesta/bin/v-list-user cmd=/usr/local/vesta/bin/v-list-user-backups cmd=/usr/local/vesta/bin/v-list-user-favourites cmd=/usr/local/vesta/bin/v-list-user-ips cmd=/usr/local/vesta/bin/v-list-user-notifications cmd=/usr/local/vesta/bin/v-list-user-packages cmd=/usr/local/vesta/bin/v-list-users cmd=/usr/local/vesta/bin/v-list-users-stats cmd=/usr/local/vesta/bin/v-list-web-domain cmd=/usr/local/vesta/bin/v-list-web-domains cmd=/usr/local/vesta/bin/v-list-web-domain-ssl cmd=/usr/local/vesta/bin/v-list-web-stats cmd=/usr/local/vesta/bin/v-list-web-templates cmd=/usr/local/vesta/bin/v-list-web-templates-backend cmd=/usr/local/vesta/bin/v-restart-web cmd=/usr/local/vesta/bin/v-update-letsencrypt-ssl cmd=/usr/local/vesta/bin/v-update-sys-queue cmd=/usr/local/vesta/bin/v-update-sys-rrd cmd=/usr/local/vesta/bin/v-update-sys-vesta-all cmd=/usr/local/vesta/bin/v-update-user-stats
I checked arguments to every command, they were all OK, no traces of injection attempts etc.
This makes me think that infection took place from root account (by default, auditd does not log commands run by root): if VestaCP were vulnerable, I think I would see something suspicious either in audit.log or in secure / auth.log. I didn't, therefore either the infection took place long ago (I have logs back to Mar 13) -- but in this case someone should have noticed something strange -- libudev.so is detected even by ClamAV, or the infection was performed with root privileges. If so, need to find what was in common between the affected servers. So far, I know that they all run VestaCP (I heard of infected Debian servers). For infection to happen via root account, it either needs to be downloaded and run by the user itself (does anybody verify the checksum of the Vesta installer?), or come from the repository. This can explain the fact that there are no new infections — old (compromised?) files were overwritten with new ones.
I would recommend the VestaCP team to take a look at their infrastructure and see if there are any signs of compromise / unauthorized access.
Check this, there are many people not able to install fron scratch.
https://www.digitalocean.com/community/ ... l-8th-2018