We are happy to announce that Vesta is back under active development as of 25 February 2024. We are working on v1 candidate and expect to engage more with the community over the coming months. We are committed to open source, and we encourage contributors to help us build the future of Vesta.
Apache not releasing memory
Apache not releasing memory
Whilst sound asleep, a WordPress installation of mine hosted on Vesta 0.9.8-17 with Apache2 + nginx had it's xmlrpc.php file totally hammered by some bot somewhere for a few hours.
There's thousands of requests like the one below - almost 4 a second.
NixStats had Apache's memory usage at ~93%. Server has 2GB of RAM.
When I blocked the IP, the requests immediately stopped and so did the bot's connection attempts.
Apache's memory usage however did not come down.
I monitored it for 30 mins and am certain that the offending IP or any other IP did was not attempting to connect, but memory usage still did not come down.
I had to restart Apache for it to free up all the RAM it was using.
If there are no requests, why didn't apache's memory usage come down on it's own?
There's thousands of requests like the one below - almost 4 a second.
Code: Select all
69.197.x.x - - [05/Dec/2017:05:12:53 +0000] "POST /xmlrpc.php HTTP/1.0" 200 926 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"
When I blocked the IP, the requests immediately stopped and so did the bot's connection attempts.
Apache's memory usage however did not come down.
I monitored it for 30 mins and am certain that the offending IP or any other IP did was not attempting to connect, but memory usage still did not come down.
I had to restart Apache for it to free up all the RAM it was using.
If there are no requests, why didn't apache's memory usage come down on it's own?
Re: Apache not releasing memory
I think the default apache configuration is 'prefork'. In which case you might like to play with the following area of your /etc/apache2/apache2.conf file.
https://httpd.apache.org/docs/2.4/mod/prefork.html
Seems to me, unless you have a very busy server, these defaults are generous. With the default settings above, Apache is running at 350Mb on a fairly lightly used server (less than 1Gb of web traffic per day).
Look around for guidance on what these actually do, but you could try halving these values for a start and see if that makes a difference.
And of course do systemctl restart apache2.service if you make any changes.
Code: Select all
<IfModule mpm_prefork_module>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 200
MaxRequestsPerChild 4000
</IfModule>
Seems to me, unless you have a very busy server, these defaults are generous. With the default settings above, Apache is running at 350Mb on a fairly lightly used server (less than 1Gb of web traffic per day).
Look around for guidance on what these actually do, but you could try halving these values for a start and see if that makes a difference.
Code: Select all
<IfModule mpm_prefork_module>
StartServers 4
MinSpareServers 2
MaxSpareServers 8
ServerLimit 128
MaxClients 50
MaxRequestsPerChild 2000
</IfModule>
Re: Apache not releasing memory
Another thought ... you might just want to block xmlrpc.php in your .htaccess file.
Re: Apache not releasing memory
plutocrat thanks for replying.
I have a good idea of how to ensure that Apache doesn't utilise too much memory and start too many child processes.
I was actually trying to figure out why Apache didn't release any of the memory it was using once the attack had stopped and why it had to be restarted to free up those resources?
I have a good idea of how to ensure that Apache doesn't utilise too much memory and start too many child processes.
I was actually trying to figure out why Apache didn't release any of the memory it was using once the attack had stopped and why it had to be restarted to free up those resources?
I usually get fail2ban to keep an eye on these things :)plutocrat wrote:Another thought ... you might just want to block xmlrpc.php in your .htaccess file.
Last edited by hwcltjn on Thu Dec 07, 2017 11:34 am, edited 1 time in total.
Re: Apache not releasing memory
OK, a couple more thoughts.
First of all, what is happening to your swap at the time of the attack? And do you have any OOM messages in your /var/log/syslog (or messages or kern.log). If mem and swap fills up completely, then yes, apps can get stuck. Maybe running monit might help you in this case .. it can help you automatically restart processes.
The MaxRequestsPerChild directive might also help you. Lowering this would restart Apache child processes more frequently.
What values do you have for MaxMemFree?
In your PHP config, what do you have for max_execution_time.
If you're really using fail2ban to track xmlrpc.php, then its not doing a very good job! ;-) Seriously I haven't actually found that completely blocking access to it has any downside, so you might as well do that. But if you don't want to, you can rate limit it with a webserver config.
eg. https://medium.com/@tturnbull/throttle- ... c4a12b7f76
First of all, what is happening to your swap at the time of the attack? And do you have any OOM messages in your /var/log/syslog (or messages or kern.log). If mem and swap fills up completely, then yes, apps can get stuck. Maybe running monit might help you in this case .. it can help you automatically restart processes.
The MaxRequestsPerChild directive might also help you. Lowering this would restart Apache child processes more frequently.
What values do you have for MaxMemFree?
In your PHP config, what do you have for max_execution_time.
If you're really using fail2ban to track xmlrpc.php, then its not doing a very good job! ;-) Seriously I haven't actually found that completely blocking access to it has any downside, so you might as well do that. But if you don't want to, you can rate limit it with a webserver config.
eg. https://medium.com/@tturnbull/throttle- ... c4a12b7f76
Re: Apache not releasing memory
Thanks plutocrat - really appreciate the guidance.
I've been keeping an eye on Apache and memory is hovering on average at 75% usage at the moment.
I'll keep monitoring and reading about it quickly think that I will add it.
I've been keeping an eye on Apache and memory is hovering on average at 75% usage at the moment.
According to NixStats, about 300Mb of swap was used during the attack. Upon Apache restart it dropped to 102Mb.plutocrat wrote: First of all, what is happening to your swap at the time of the attack?
No OOM messages.plutocrat wrote: And do you have any OOM messages in your /var/log/syslog (or messages or kern.log). If mem and swap fills up completely, then yes, apps can get stuck.
Thanks, I've lowered this from 4,000 to 2,000 - let's see how it plays out.plutocrat wrote: The MaxRequestsPerChild directive might also help you. Lowering this would restart Apache child processes more frequently.
I have no value for MaxMemFree in Apache conf - so none I guess.plutocrat wrote: What values do you have for MaxMemFree?
I'll keep monitoring and reading about it quickly think that I will add it.
Set to 30 - default.plutocrat wrote: In your PHP config, what do you have for max_execution_time.
It was a dev server, so I hadn't enabled it yet ><plutocrat wrote: If you're really using fail2ban to track xmlrpc.php, then its not doing a very good job! ;-)