what does "upstream timed out" error mean?

[baijianpeng]

My Joomla website is running on VestaCP on CentOS 7. The PHP version is 5.6.21, which was installed with "PHP Selector" from skamasle.

Today I checked the log file /var/log/httpd/domains/joomlagate.com.error.log, there are following error records:

2016/05/15 06:08:51 [error] 30728#30728: *1920053 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 42.236.48.92, server: joomlagate.com, request: "GET /index.php HTTP/1.1", upstream: "http://120.27.137.71:8080/index.php", host: "www.joomlagate.com", referrer: "http://www.joomlagate.com/index.php"
2016/05/15 06:08:51 [error] 30728#30728: *1920063 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 220.181.108.162, server: joomlagate.com, request: "GET /index.php?option=com_kunena&view=category&defaultmenu=155&Itemid=154&catid=46&limitstart=20 HTTP/1.1", upstream: "http://120.27.137.71:8080/index.php?opt ... itstart=20", host: "www.joomlagate.com"
2016/05/15 06:09:08 [error] 30729#30729: *1919841 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 59.127.53.104, server: joomlagate.com, request: "POST /?option=com_komento HTTP/2.0", upstream: "https://120.27.137.71:8443/?option=com_komento", host: "www.joomlagate.com", referrer: "https://www.joomlagate.com/"
2016/05/15 06:09:08 [error] 30729#30729: *1919841 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 59.127.53.104, server: joomlagate.com, request: "POST /?option=com_komento HTTP/2.0", upstream: "https://120.27.137.71:8443/?option=com_komento", host: "www.joomlagate.com", referrer: "https://www.joomlagate.com/"
2016/05/15 06:09:08 [error] 30729#30729: *1919844 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 117.136.0.91, server: joomlagate.com, request: "POST /?option=com_komento HTTP/2.0", upstream: "https://120.27.137.71:8443/?option=com_komento", host: "www.joomlagate.com", referrer: "https://www.joomlagate.com/"
2016/05/15 06:09:08 [error] 30729#30729: *1920045 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 106.120.85.34, server: joomlagate.com, request: "POST /?option=com_komento HTTP/2.0", upstream: "https://120.27.137.71:8443/?option=com_komento", host: "www.joomlagate.com", referrer: "https://www.joomlagate.com/"
2016/05/15 06:09:08 [error] 30729#30729: *1920045 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 106.120.85.34, server: joomlagate.com, request: "POST /?option=com_komento HTTP/2.0", upstream: "https://120.27.137.71:8443/?option=com_komento", host: "www.joomlagate.com", referrer: "https://www.joomlagate.com/"
2016/05/15 06:09:08 [error] 30729#30729: *1920056 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 203.208.60.233, server: joomlagate.com, request: "GET /index.php?option=com_kunena&view=topic&catid=59&id=61436&layout=unread&Itemid=154 HTTP/1.1", upstream: "http://120.27.137.71:8080/index.php?opt ... Itemid=154", host: "joomlagate.com"
2016/05/15 06:50:53 [crit] 30728#30728: *1921989 SSL_do_handshake() failed (SSL: error:14094085:SSL routines:SSL3_READ_BYTES:ccs received early) while SSL handshaking, client: 64.41.200.103, server: 120.27.137.71:443
2016/05/15 06:50:53 [crit] 30728#30728: *1921990 SSL_do_handshake() failed (SSL: error:14094085:SSL routines:SSL3_READ_BYTES:ccs received early) while SSL handshaking, client: 64.41.200.103, server: 120.27.137.71:443

What does "upstream timed out" error mean? How to fix this?

Thank you.

[SS88]

That means it takes your web server more than 60 seconds to respond. To solve this problem in nginx.conf change the value of proxy_read_timeout directive. This directive determines how long nginx will wait to get the response to a request. By default it's 60 seconds. Change it to 300 seconds:

Code: Select all

proxy_read_timeout 300;

ref: http://howtounix.info/howto/110-connect ... r-in-nginx

[baijianpeng]

hi,@SS88, your instruction worked. After I insert that line to the "server" section of my snginx.conf (I use HTTPS) , there is no more "upstream timed out" errors recorded in the log file.

However, now I got following new errors, could you please give me further help to solve this?

[Sun May 15 08:26:54.946756 2016] [fcgid:warn] [pid 25306] [client 124.88.120.128:44564] mod_fcgid: read data timeout in 120 seconds, referer: https://www.joomlagate.com/index.php?op ... &Itemid=19
[Sun May 15 08:26:54.946818 2016] [core:error] [pid 25306] [client 124.88.120.128:44564] End of script output before headers: index.php, referer: https://www.joomlagate.com/index.php?op ... &Itemid=19
[Sun May 15 08:33:11.776640 2016] [fcgid:warn] [pid 25308] [client 101.226.166.218:44846] mod_fcgid: read data timeout in 120 seconds, referer: https://120.27.137.71/index.php?Itemid= ... w=category
[Sun May 15 08:33:11.776699 2016] [core:error] [pid 25308] [client 101.226.166.218:44846] End of script output before headers: index.php, referer: https://120.27.137.71/index.php?Itemid= ... w=category
[Sun May 15 08:42:05.375294 2016] [fcgid:warn] [pid 25009] [client 113.240.234.205:45175] mod_fcgid: read data timeout in 120 seconds, referer: https://www.joomlagate.com/
[Sun May 15 08:42:05.375362 2016] [core:error] [pid 25009] [client 113.240.234.205:45175] End of script output before headers: index.php, referer: https://www.joomlagate.com/

Thank you.

[SS88]

Change these to all 300.

Code: Select all

keepalive_timeout 300;
proxy_read_timeout 300;
proxy_connect_timeout 300;
fastcgi_read_timeout 300;

300 = 5 minutes

If your pages are not loading within 5 minutes I would look into why. I had a browse and pages were loading for me quite quickly so you may want to take a look as to why these pages are not loading within 120 seconds (as that's a very long time to wait)

You also may need to increase FcgidIOTimeout to 300 (this setting is a Fast CGI setting)

[baijianpeng]

Today I met "upstream timed out" error again on my re-installed VPS server. So I followed your above post to change those timing values.

Well, I found all those 4 parameters in this file: /etc/nginx/nginx.conf.

However, the format of "keepalive_timeout" directive is not a single value, but a pair :

Code: Select all

keepalive_timeout               60 60;

This is the default value of that line.

What kind of new value should I change to for this line?

Thank you.

[SS88]

Today I met "upstream timed out" error again on my re-installed VPS server. So I followed your above post to change those timing values.

Well, I found all those 4 parameters in this file: /etc/nginx/nginx.conf.

However, the format of "keepalive_timeout" directive is not a single value, but a pair :

Code: Select all

keepalive_timeout               60 60;

This is the default value of that line.

What kind of new value should I change to for this line?

Thank you.

Syntax: keepalive_timeout timeout [header_timeout];
Default: keepalive_timeout 75s;
Context: http, server, location

The first parameter sets a timeout during which a keep-alive client connection will stay open on the server side. The zero value disables keep-alive client connections. The optional second parameter sets a value in the “Keep-Alive: timeout=time” response header field. Two parameters may differ.

The “Keep-Alive: timeout=time” header field is recognized by Mozilla and Konqueror. MSIE closes keep-alive connections by itself in about 60 seconds.


http://nginx.org/en/docs/http/ngx_http_ ... ve_timeout

[ashleycalebhart]

May i ask why you would set a tmeout limit of 5 minutes? asking for DDos troubles here?

[SS88]

May i ask why you would set a tmeout limit of 5 minutes? asking for DDos troubles here?

If you're under a real DDoS you have no chance anyway.

[baijianpeng]

Well, I tried to change both number to "300" for the "keepalive_timeout" parameter, then I saved that conf file and tried to restart Nginx, then I got following error:

Code: Select all

root@mail:/etc/nginx# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.

root@mail:/etc/nginx# systemctl status nginx
● nginx.service - LSB: Stop/start nginx
   Loaded: loaded (/etc/init.d/nginx; bad; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2017-01-28 08:28:44 CST; 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6286 ExecStop=/etc/init.d/nginx stop (code=exited, status=0/SUCCESS)
  Process: 6325 ExecStart=/etc/init.d/nginx start (code=exited, status=1/FAILURE)
   CGroup: /system.slice/nginx.service
           ├─25307 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.con
           ├─25308 nginx: worker process                   
           └─25309 nginx: cache manager process            

Jan 28 08:28:43 mail.joomlagate.com nginx[6325]: nginx: [emerg] bind() to 120.27.137.71:443 failed (98: Address a
Jan 28 08:28:43 mail.joomlagate.com nginx[6325]: nginx: [emerg] bind() to 10.47.93.1:80 failed (98: Address alrea
Jan 28 08:28:43 mail.joomlagate.com nginx[6325]: nginx: [emerg] bind() to 120.27.137.71:80 failed (98: Address al
Jan 28 08:28:43 mail.joomlagate.com nginx[6325]: nginx: [emerg] bind() to 127.0.0.1:8084 failed (98: Address alre
Jan 28 08:28:43 mail.joomlagate.com nginx[6325]: nginx: [emerg] bind() to 120.27.137.71:443 failed (98: Address a
Jan 28 08:28:44 mail.joomlagate.com nginx[6325]: nginx: [emerg] still could not bind()
Jan 28 08:28:44 mail.joomlagate.com systemd[1]: nginx.service: Control process exited, code=exited status=1
Jan 28 08:28:44 mail.joomlagate.com systemd[1]: Failed to start LSB: Stop/start nginx.
Jan 28 08:28:44 mail.joomlagate.com systemd[1]: nginx.service: Unit entered failed state.
Jan 28 08:28:44 mail.joomlagate.com systemd[1]: nginx.service: Failed with result 'exit-code'.
lines 1-21/21 (END)

At the same time, I checked the Nginx log files and found following records:

root@mail:/var/log/nginx# tail error.log
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 120.27.137.71:443 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 10.47.93.1:80 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 120.27.137.71:80 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 127.0.0.1:8084 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 120.27.137.71:443 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 10.47.93.1:80 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 120.27.137.71:80 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 127.0.0.1:8084 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: bind() to 120.27.137.71:443 failed (98: Address already in use)
2017/01/28 08:28:41 [emerg] 6331#6331: still could not bind()

Now my Nginx can NOT be restarted, even after I change those numbers back to "60 60".

But, my website pages can still be opened. This is weird!

[baijianpeng]

Ok, since I am not a pro , so I choose to "reboot" my server. After reboot, Nginx seems OK now.

Code: Select all

root@mail:~# systemctl status nginx
● nginx.service - LSB: Stop/start nginx
   Loaded: loaded (/etc/init.d/nginx; bad; vendor preset: enabled)
   Active: active (running) since Sat 2017-01-28 08:37:55 CST; 1min 35s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 783 ExecStart=/etc/init.d/nginx start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nginx.service
           ├─1037 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.con
           ├─1038 nginx: worker process                   
           └─1039 nginx: cache manager process            

Jan 28 08:37:54 mail.joomlagate.com systemd[1]: Starting LSB: Stop/start nginx...
Jan 28 08:37:55 mail.joomlagate.com systemd[1]: Started LSB: Stop/start nginx.