Apache not releasing memory

[hwcltjn]

Whilst sound asleep, a WordPress installation of mine hosted on Vesta 0.9.8-17 with Apache2 + nginx had it's xmlrpc.php file totally hammered by some bot somewhere for a few hours.

There's thousands of requests like the one below - almost 4 a second.

Code: Select all

69.197.x.x - - [05/Dec/2017:05:12:53 +0000] "POST /xmlrpc.php HTTP/1.0" 200 926 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"

NixStats had Apache's memory usage at ~93%. Server has 2GB of RAM.
When I blocked the IP, the requests immediately stopped and so did the bot's connection attempts.

Apache's memory usage however did not come down.
I monitored it for 30 mins and am certain that the offending IP or any other IP did was not attempting to connect, but memory usage still did not come down.

I had to restart Apache for it to free up all the RAM it was using.

If there are no requests, why didn't apache's memory usage come down on it's own?

[plutocrat]

I think the default apache configuration is 'prefork'. In which case you might like to play with the following area of your /etc/apache2/apache2.conf file.

Code: Select all

<IfModule mpm_prefork_module>
    StartServers          8
    MinSpareServers       5
    MaxSpareServers      20
    ServerLimit         256
    MaxClients          200
    MaxRequestsPerChild 4000
</IfModule>

https://httpd.apache.org/docs/2.4/mod/prefork.html
Seems to me, unless you have a very busy server, these defaults are generous. With the default settings above, Apache is running at 350Mb on a fairly lightly used server (less than 1Gb of web traffic per day).
Look around for guidance on what these actually do, but you could try halving these values for a start and see if that makes a difference.

Code: Select all

<IfModule mpm_prefork_module>
    StartServers          4
    MinSpareServers       2
    MaxSpareServers       8
    ServerLimit         128
    MaxClients           50
    MaxRequestsPerChild 2000
</IfModule>

And of course do systemctl restart apache2.service if you make any changes.

[plutocrat]

Another thought ... you might just want to block xmlrpc.php in your .htaccess file.

[hwcltjn]

plutocrat thanks for replying.
I have a good idea of how to ensure that Apache doesn't utilise too much memory and start too many child processes.

I was actually trying to figure out why Apache didn't release any of the memory it was using once the attack had stopped and why it had to be restarted to free up those resources?

Another thought ... you might just want to block xmlrpc.php in your .htaccess file.

I usually get fail2ban to keep an eye on these things :)

[plutocrat]

OK, a couple more thoughts.
First of all, what is happening to your swap at the time of the attack? And do you have any OOM messages in your /var/log/syslog (or messages or kern.log). If mem and swap fills up completely, then yes, apps can get stuck. Maybe running monit might help you in this case .. it can help you automatically restart processes.

The MaxRequestsPerChild directive might also help you. Lowering this would restart Apache child processes more frequently.

What values do you have for MaxMemFree?

In your PHP config, what do you have for max_execution_time.

If you're really using fail2ban to track xmlrpc.php, then its not doing a very good job! ;-) Seriously I haven't actually found that completely blocking access to it has any downside, so you might as well do that. But if you don't want to, you can rate limit it with a webserver config.
eg. https://medium.com/@tturnbull/throttle- ... c4a12b7f76

[hwcltjn]

Thanks plutocrat - really appreciate the guidance.
I've been keeping an eye on Apache and memory is hovering on average at 75% usage at the moment.

First of all, what is happening to your swap at the time of the attack?

According to NixStats, about 300Mb of swap was used during the attack. Upon Apache restart it dropped to 102Mb.

And do you have any OOM messages in your /var/log/syslog (or messages or kern.log). If mem and swap fills up completely, then yes, apps can get stuck.

No OOM messages.

The MaxRequestsPerChild directive might also help you. Lowering this would restart Apache child processes more frequently.

Thanks, I've lowered this from 4,000 to 2,000 - let's see how it plays out.

What values do you have for MaxMemFree?

I have no value for MaxMemFree in Apache conf - so none I guess.
I'll keep monitoring and reading about it quickly think that I will add it.

In your PHP config, what do you have for max_execution_time.

Set to 30 - default.

If you're really using fail2ban to track xmlrpc.php, then its not doing a very good job! ;-)

It was a dev server, so I hadn't enabled it yet ><