F*CK !!! Please help me TROUBLESHOOT public_html/megla.txt - "hacked by megla akash from Team_CC" Topic is solved
F*CK !!! Please help me TROUBLESHOOT public_html/megla.txt - "hacked by megla akash from Team_CC"
I currently have two servers with VestaCP. My dev server (18+ months up & running) and my newest production server (2+ months old). Both running Ubuntu 16.04 LTS with NGINX as a webserver but with different VPS providers. And the 2nd / my latest server just got hit by a hacker, and it got hit bad?! Not sure yet.
Literally found out about this 10minutes ago. :-(
As that megla.txt with file contents:
I'm not only puzzled how it could have happened but also a bit scared what else got uploaded or has been modified on the server files and/or DB wise. Even more so, as I use strong & unique passwords, everything running at custom ports (both SSH as well as VestaCP admin), all sites have LetsEncrypt SSL certificates + CloudFlare, and on a WP levels got advanced & hardened iThemes security running (REST/XML-RPC disabled, no execution of PHP scripts in themes / uploads, long string filtering, illegal character filtering in parameters) with either Wordfence Security or JetPack as a second line of defense, moreover have centralized management (all sites are always up-to-date with MainWP over SSL). It can't be they got access to my MainWP dashboard, as other things on different servers should have been affected then. And running end-point enterprise internet security on my workstations + a diversity of firewall solutions + pi-hole. Haven't used public wifi or anything of sorts either. What else? No crazy chmods, chowns, and `su` has to be used to gain root access.
In other words: some help suggestions / insights would be appreciated, how to troubleshoot + fix this properly + prevent it from happening next time. While I'm going to ... not really sure where to start. Hence using the F-word in the title. I'm not a newbie, so erhhhh .. its either something really silly that I overlooked, or those hackers are truly skilled; WTF?!
Progress report / things I've done (keeping this up-to-date by the hour .. with or without replies):
+ removed all megla.txt files. So I don't get indexed / traced as hacked
+ changed password of VestaCP admin user (although it was setup in VestaCP firewall restricted to 1 single ip)
+ ran `clamscan -r -i /home` from the KVM. Result: no infected files
+ chkrootkit found nothing out of the ordinary either
+ MainWP's suruci sweep on all sites found a few things, but nothing major
+ ...
+ ..
+ ?
The Culprit
I think I figured it out what happened ('only' took 7+ hours to figure out! LOL). I also had two WP Multi-User staging projects running on VestaCP admin account. Imported from a prior shared hosting account, and not hooked into MainWP (as that feature doesn't exist). Aka not up-to-date, neither having a lot of WP hardening in it, as that's tough to do on Multi-User environment without WPMUdev subscription (something I should have had; in hindsight). Thus I probably got sql hijacked based on old plugins, either 4.7x WP version on the WPMU projects, and from there on the entire VestaCP account got infected? That's at least my best guess thus far. Should have kept track of the timestamps in the order those megla.txt files were created. Hmz. *update* found a bug in VestaCP (again) combined with NGINX. Wordfence uses .user.ini to create the WP WAF. That -- supposedly hidden file -- is downloadable on a NGINX server. Makes me wonder what other typical LAMP stack files are publicly accessible on complex WP environments. Scary!
How to fix
Well .. can't spend too much time contemplating on the cause this, except how I will go over VestaCP's LEMP webserver templates (again). Thereafter I'm just going to export all the pages, posts, etc. into XML files. Then delete the "admin" VestaCP account, as there seems to be no need to reinstall the server, create a new 'admin' account and build everything up from a WP point a view. Thats going to be an intensive weekend. ;-) But seems to be the safest option, although the VestaCP backups from yesterday seem unaffected.
How to prevent it
Divided projects over more VestaCP users, and definitely put WPMU projects apart from single installs. And next to all the other security measures I had in play, also have regular audits with that tool shared in the next post. No pain; no gain. ;-)
Literally found out about this 10minutes ago. :-(
As that megla.txt with file contents:
showed up in every public_html on my VestaCP powered server of default VestaCP user "admin". Sadly, have 11 HUGE WP projects for that user with 100+ WP plugins per project. A second VestaCP user on my server with 3 other WP sites, didn't seem to get affected. But that's just based on the premise it didn't contain megla.txt files (as I ran a `find / -name "megla.txt" > results.txt` on the server).hacked by megla akash from Team_CC
I'm not only puzzled how it could have happened but also a bit scared what else got uploaded or has been modified on the server files and/or DB wise. Even more so, as I use strong & unique passwords, everything running at custom ports (both SSH as well as VestaCP admin), all sites have LetsEncrypt SSL certificates + CloudFlare, and on a WP levels got advanced & hardened iThemes security running (REST/XML-RPC disabled, no execution of PHP scripts in themes / uploads, long string filtering, illegal character filtering in parameters) with either Wordfence Security or JetPack as a second line of defense, moreover have centralized management (all sites are always up-to-date with MainWP over SSL). It can't be they got access to my MainWP dashboard, as other things on different servers should have been affected then. And running end-point enterprise internet security on my workstations + a diversity of firewall solutions + pi-hole. Haven't used public wifi or anything of sorts either. What else? No crazy chmods, chowns, and `su` has to be used to gain root access.
In other words: some help suggestions / insights would be appreciated, how to troubleshoot + fix this properly + prevent it from happening next time. While I'm going to ... not really sure where to start. Hence using the F-word in the title. I'm not a newbie, so erhhhh .. its either something really silly that I overlooked, or those hackers are truly skilled; WTF?!
Progress report / things I've done (keeping this up-to-date by the hour .. with or without replies):
+ removed all megla.txt files. So I don't get indexed / traced as hacked
+ changed password of VestaCP admin user (although it was setup in VestaCP firewall restricted to 1 single ip)
+ ran `clamscan -r -i /home` from the KVM. Result: no infected files
+ chkrootkit found nothing out of the ordinary either
+ MainWP's suruci sweep on all sites found a few things, but nothing major
+ ...
+ ..
+ ?
The Culprit
I think I figured it out what happened ('only' took 7+ hours to figure out! LOL). I also had two WP Multi-User staging projects running on VestaCP admin account. Imported from a prior shared hosting account, and not hooked into MainWP (as that feature doesn't exist). Aka not up-to-date, neither having a lot of WP hardening in it, as that's tough to do on Multi-User environment without WPMUdev subscription (something I should have had; in hindsight). Thus I probably got sql hijacked based on old plugins, either 4.7x WP version on the WPMU projects, and from there on the entire VestaCP account got infected? That's at least my best guess thus far. Should have kept track of the timestamps in the order those megla.txt files were created. Hmz. *update* found a bug in VestaCP (again) combined with NGINX. Wordfence uses .user.ini to create the WP WAF. That -- supposedly hidden file -- is downloadable on a NGINX server. Makes me wonder what other typical LAMP stack files are publicly accessible on complex WP environments. Scary!
How to fix
Well .. can't spend too much time contemplating on the cause this, except how I will go over VestaCP's LEMP webserver templates (again). Thereafter I'm just going to export all the pages, posts, etc. into XML files. Then delete the "admin" VestaCP account, as there seems to be no need to reinstall the server, create a new 'admin' account and build everything up from a WP point a view. Thats going to be an intensive weekend. ;-) But seems to be the safest option, although the VestaCP backups from yesterday seem unaffected.
How to prevent it
Divided projects over more VestaCP users, and definitely put WPMU projects apart from single installs. And next to all the other security measures I had in play, also have regular audits with that tool shared in the next post. No pain; no gain. ;-)
Last edited by rhyker2u on Sat Mar 17, 2018 3:12 pm, edited 34 times in total.
Re: F*CK !!! Please help me TROUBLESHOOT public_html/megla.txt - "hacked by megla akash from Team_CC"
What's the best Linux AV file scanner? *edit* already running a few. Didn't know about Lynis yet: https://www.digitalocean.com/community/ ... untu-16-04 *edit2* == MAGIC; saved a ton of time!
I hope someone else can benefit from this info.
I hope someone else can benefit from this info.
Re: F*CK !!! Please help me TROUBLESHOOT public_html/megla.txt - "hacked by megla akash from Team_CC"
Thanks for the tip.
Going to try Lynis with my centos7
Going to try Lynis with my centos7
Re: F*CK !!! Please help me TROUBLESHOOT public_html/megla.txt - "hacked by megla akash from Team_CC"
potential problems are no problem ;-) What IS a real-world problem, is the default wordpress2.tpl and wordpress2.stpl of VestaCP. Here's the up-to-date solution: viewtopic.php?f=11&t=13668&p=68082#p68082
And just in case you wonder on the performance impact of that many plugins? :-p Never was a problem with a properly configured VestaCP environment either. But they only recently switched from dynamic to on-demand PHP-FPM (although still configured wrong). Will do an update to [GUIDE] WordPress at Ubuntu 16.04 NGINX PHP(7)-FPM + Redis Object caching FIX == FTL about that too. As doing it through Apache? Oh yeah that was an interesting experience for sure :) Although with a LAMP stack I probably would not have been hacked. Oh well, thankfully this happened prior to full-blown launch of all the hosted projects.
And just in case you wonder on the performance impact of that many plugins? :-p Never was a problem with a properly configured VestaCP environment either. But they only recently switched from dynamic to on-demand PHP-FPM (although still configured wrong). Will do an update to [GUIDE] WordPress at Ubuntu 16.04 NGINX PHP(7)-FPM + Redis Object caching FIX == FTL about that too. As doing it through Apache? Oh yeah that was an interesting experience for sure :) Although with a LAMP stack I probably would not have been hacked. Oh well, thankfully this happened prior to full-blown launch of all the hosted projects.