Page 1 of 3
Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Tue Feb 10, 2015 1:51 am
by Galaxian
I'm kind of dumbfounded with this one, probably because I'm a total noob when it comes to server management beyond the control panel.
I use Google PageSpeed Service and it works well, until the Facebook scraper tries to load a page and gets a 500 internal server error thrown back at it. This is a problem for my live and actively shared sites.
Does anyone know what'd cause these conditions to return a 500 for the Facebook scraper (and debugger tool):
- Only happens only htaccess URL rewritten pages (also including the root /, but trying index.php directly works)
- Only happens on sites using Google PageSpeed Service. Yes, I've tried deactivating the rewriters, but it doesn't seem to be any of those.
- Works when using nginx's caching template. None of the others work. Also unticking "nginx support" doesn't change anything. However, I can't really keep this active because I have dynamic features that break when this is activated.
Remember, the sites seem to load fine otherwise. This problem only occurs for the FB scraper AFAIK. Any suggestions?
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Thu Feb 12, 2015 5:49 pm
by skurudo
I don't understand, what's facebook debugger or scrapper, but I think it some bot.
You need block this bot? Then block it with nginx. You need just identify this bot from logs and made changes in nginx config.
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Fri Feb 13, 2015 1:52 am
by Galaxian
skurudo wrote:I don't understand, what's facebook debugger or scrapper, but I think it some bot.
You need block this bot? Then block it with nginx. You need just identify this bot from logs and made changes in nginx config.
When I say the Facebook scraper, I mean requests from Facebook. You know when you share something on Facebook, you get a little image, a title, and description on the news feed? Basically, I am talking about the method Facebook uses to obtain that data from the page. When I receive a request like that from Facebook, it gets a 500 internal server error.
By the way, the Facebook debug tool can be used from here:
https://developers.facebook.com/tools/debug/
It can force a re-fetch of the page data. I do not need to block it, I need to know what can cause nginx to throw a 500 error with those specific conditions that I mentioned in my original post. I know that it's hard to understand.
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Fri Feb 13, 2015 5:36 am
by skurudo
It's same. You chose, what stutus return 403/404 or 500
Code: Select all
location / {
if ($http_user_agent ~ (libwww|Wget|LWP|damnBot|BBBike|java|spider|crawl) ) {
return 500;
}
}
You just need identify facebook user agent in logs
And then add something like thin in nginx config for your domain for location /
Code: Select all
{
if ($http_user_agent ~ (FacebookExternalHit) ) {
return 500;
}
}
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Fri Feb 13, 2015 5:34 pm
by Galaxian
You're missing the point completely. I don't want to block the Facebook bot, I want to figure out why it is receiving an internal server error when it tries to load the page under those conditions I stated in the original post.
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Sat Feb 14, 2015 1:09 am
by skurudo
Galaxian wrote:You're missing the point completely. I don't want to block the Facebook bot, I want to figure out why it is receiving an internal server error when it tries to load the page under those conditions I stated in the original post.
It's clear now. For this I see only one way - explore logs. Search, when facebook come, and see, what error was in your scripts. There must be something.
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Sat Feb 28, 2015 7:43 pm
by Galaxian
Thanks for your response. Forgive me for my noobishness, but which logs?
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Sat Feb 28, 2015 10:40 pm
by joem
Galaxian wrote:Thanks for your response. Forgive me for my noobishness, but which logs?
Code: Select all
nano $VESTA/data/templates/web/nginx/default.tpl
Under
Code: Select all
error_log /var/log/%web_system%/domains/%domain%.error.log error;
location / {
proxy_pass http://%ip%:%web_port%;
Add
Code: Select all
proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;
If that doesnt work you can also try this,
Right below
Code: Select all
location ~* ^.+\.(%proxy_extentions%)$ {
Add
Make sure after you save all your changes you rebuild the users web you can do this by,
Login To Vesta CP -> Click Users Tab -> Toggle All -> In the drop down menu select rebuild web
Also restart nginx
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Wed Mar 04, 2015 5:43 pm
by Galaxian
joem wrote:Galaxian wrote:Thanks for your response. Forgive me for my noobishness, but which logs?
Code: Select all
nano $VESTA/data/templates/web/nginx/default.tpl
Under
Code: Select all
error_log /var/log/%web_system%/domains/%domain%.error.log error;
location / {
proxy_pass http://%ip%:%web_port%;
Add
Code: Select all
proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;
If that doesnt work you can also try this,
Right below
Code: Select all
location ~* ^.+\.(%proxy_extentions%)$ {
Add
Make sure after you save all your changes you rebuild the users web you can do this by,
Login To Vesta CP -> Click Users Tab -> Toggle All -> In the drop down menu select rebuild web
Also restart nginx
I've done this, but I have no idea what I've just done or how it pertains to my question regarding error logs. Could you provide an explanation?
Re: Facebook Debugger gets Error 500 On Rewritten URLs
Posted: Wed Mar 04, 2015 7:35 pm
by skurudo
/var/log/apache2 or httpd
this logs