Got 10 VestaCP servers exploited

[romWeb]

sending email from my webserver (as always) but nothing , only can receive

[romWeb]

Ok, anyway thanks for your help

[kobo1d]

n0x, if you havent so yet, you can check out this poll and fill in your infos there:

Hi Everyone,

We have put together a survey to help us better understand the general configuration in relation to some of the working theories. If you have suggestions to broaden the survey, please let us know.

https://goo.gl/forms/qXtzd6nZFrKNw7DN2

We greatly appreciate any input.

[n0x]

n0x, if you havent so yet, you can check out this poll and fill in your infos there:

Hi Everyone,

We have put together a survey to help us better understand the general configuration in relation to some of the working theories. If you have suggestions to broaden the survey, please let us know.

https://goo.gl/forms/qXtzd6nZFrKNw7DN2

We greatly appreciate any input.

Didn't see that - Have completed it with info from the first time and today.

[yoko eagle]

For whom that not yet resolved this problem in DIGITAL OCEAN, here are my steps to resolved my own way without any reply from the support team. We do understand in this kind of situation they have a huge backlog in their ticketing system.

1. When your droplet got taken down, you only can access their console which is very slow and with limitation. No copy paste or keyboard shortcut, only typing allowed.

2. The first thing to do is try to clean the virus from the console. Follow carefully steps in this page:
https://admin-ahead.com/forum/server-se ... ts-trojan/
If you follow it correctly, It will stop virus reinfected. Note that the virus process name in top command will be vary and never be the same in each attack, so you have to perform you own search to any suspected files.

3. After try to clean the virus, make a snapshot. It will take approximately 30 minutes to finished due to the suspended status. Then clone the droplet.

4. Immediately create a DO firewall for the clone to stop all outbound traffic. With no outbound the droplet will not be taken down again if your first cleaning is not completed. This also give you full access via ssh and more time to perform cleaning and deep inspection.

5. After make sure the virus removed, open the outbound traffic and wait at least 1 hour. In my case I got taken down 2 times previously. Then make a backup

Code: Select all

v-backup-user admin

6. Create new droplet with same specification and install fresh vestacp with clamav. Apply the new security patch.
In my case I use debian 9.

Code: Select all

apt-get update && apt-get upgrade
v-update-sys-vesta-all

7. SSH back to your clone droplet and copy the backup to the new droplet.

Code: Select all

scp /backup/admin.date.tar new-server:/backup/

Then SSH to your new installed droplet and restore the backup

Code: Select all

v-restore-user admin admin.date.tar

8. Tightening your new vestacp such as change ssh port, vesta default port, additional authentication for login, close remote mysql port, etc. Then resetting all your sites with new IP dns record.

9. Done! Your server should goes live again with no data losses.

[Nipsey]

Yesterday I installed 0.9.8-20 on a fresh vultr centos 7.4 everything was fine the whole day until I ran yum update today, centos and vesta had updates so I accepted them and then the server was infected. Now I am afraid to run yum update after installing vesta. So I guess 0.9.8-20 is affected as well.

You can try this yourself, I don't know why yum update had vesta updates when I was already running version 0.9.8-20

Can you confirm that this was a root level infection and did it appear just the same as the majority of the reports here?

Yes it was

[Nipsey]

First of all, there was no reports about hacks on 0.9.8-20.
Please update your servers as soon as possible.


For those who are interested in technical details here is how authentication model looked like in previous releases:
- PHP script /api/index.php receives user password via POST request
- then this script writes user password to a tmp file (for example /tmp/tmp.cWdkwNbBrR)
This operation was needed to protect password from being hijacked via "ps auxf" command.

- Path to the file was then passed to the shell script v-check-user-password:
(v-check-user-password admin /tmp/tmp.cWdkwNbBrR)

- The script reads the content of /tmp/tmp.cWdkwNbBrR and calls sub process in order to generate hash based on the file content
hash=$($BIN/v-generate-password-hash $method $salt <<< $password)

We think that this part could allow for arbitrary code execution. Theoretically you could send something like
"password; cat /etc/passwd" to get the content of /etc/passwd. However we weren't able to bypass auth protection ourselves.


Here is what we did in the new release.
- The PHP process still receives unescaped password via POST
- Then instead of transmitting this password to the script it is now creates hash
- Then this hash is written into the tmp
This way code injected string like "password; cat /etc/passwd" converts to a harmless "7v8FlZefN7aQ9OoxGkR8lFHKejCxH9g64TQVVoRUuAObszO2hJy.CAs8ZG3JUtDKYQZNIZS61" sequence of characters which makes it impossible to inject anything.

Yesterday I installed 0.9.8-20 on a fresh vultr centos 7.4 everything was fine the whole day until I ran yum update today, centos and vesta had updates so I accepted them and then the server was infected. Now I am afraid to run yum update after installing vesta. So I guess 0.9.8-20 is affected as well.

You can try this yourself, I don't know why yum update had vesta updates when I was already running version 0.9.8-20

You did have Vesta at version 0.9.8-20. Now you received the update to vesta-nginx and vesta-php to those versions.

You probably already did have your server compromised before installing the updates.

How?. There was no /etc/cron.hourly/gcc.sh

[darkworks]

sending email from my webserver (as always) but nothing , only can receive

check your PTR records it your vps panel , better contact ur hosting provider for PTR records

[Felix]

imperio, skid and other VESTA Stuff,
Please consider offering an option for certificate-based access to VESTA Panel!

When enabled, only the persons with the correct certificate installed in their browsers can be presented with VESTA Panel at https://[FQDN]:8083 Then they'll need to login as usual.

Here's why I think this is a good idea

• Because as far as I understand, the attack vector isn't 100% known/reproducible
• Because it will add another strong layer of security
• Because a hacker won't login even if they've already stolen admin's credentials
• Because it bullet-proofs VESTA even if another such vulnerability is ever discovered

Please comment

[imperio]

Felix,
I'll send your suggestion to Serghey Rodin