Facebook Debugger gets Error 500 On Rewritten URLs

[Galaxian]

I'm kind of dumbfounded with this one, probably because I'm a total noob when it comes to server management beyond the control panel.

I use Google PageSpeed Service and it works well, until the Facebook scraper tries to load a page and gets a 500 internal server error thrown back at it. This is a problem for my live and actively shared sites.

Does anyone know what'd cause these conditions to return a 500 for the Facebook scraper (and debugger tool):

- Only happens only htaccess URL rewritten pages (also including the root /, but trying index.php directly works)
- Only happens on sites using Google PageSpeed Service. Yes, I've tried deactivating the rewriters, but it doesn't seem to be any of those.
- Works when using nginx's caching template. None of the others work. Also unticking "nginx support" doesn't change anything. However, I can't really keep this active because I have dynamic features that break when this is activated.

Remember, the sites seem to load fine otherwise. This problem only occurs for the FB scraper AFAIK. Any suggestions?

[skurudo]

I don't understand, what's facebook debugger or scrapper, but I think it some bot.
You need block this bot? Then block it with nginx. You need just identify this bot from logs and made changes in nginx config.

[Galaxian]

I don't understand, what's facebook debugger or scrapper, but I think it some bot.
You need block this bot? Then block it with nginx. You need just identify this bot from logs and made changes in nginx config.

When I say the Facebook scraper, I mean requests from Facebook. You know when you share something on Facebook, you get a little image, a title, and description on the news feed? Basically, I am talking about the method Facebook uses to obtain that data from the page. When I receive a request like that from Facebook, it gets a 500 internal server error.

By the way, the Facebook debug tool can be used from here: https://developers.facebook.com/tools/debug/

It can force a re-fetch of the page data. I do not need to block it, I need to know what can cause nginx to throw a 500 error with those specific conditions that I mentioned in my original post. I know that it's hard to understand.

[skurudo]

It's same. You chose, what stutus return 403/404 or 500

Code: Select all

location / {
   if ($http_user_agent ~ (libwww|Wget|LWP|damnBot|BBBike|java|spider|crawl) ) {
       return 500;
   }
}

You just need identify facebook user agent in logs
And then add something like thin in nginx config for your domain for location /

Code: Select all

   {
   if ($http_user_agent ~ (FacebookExternalHit) ) {
       return 500;
   }
}

[Galaxian]

You're missing the point completely. I don't want to block the Facebook bot, I want to figure out why it is receiving an internal server error when it tries to load the page under those conditions I stated in the original post.

[skurudo]

You're missing the point completely. I don't want to block the Facebook bot, I want to figure out why it is receiving an internal server error when it tries to load the page under those conditions I stated in the original post.

It's clear now. For this I see only one way - explore logs. Search, when facebook come, and see, what error was in your scripts. There must be something.

[Galaxian]

Thanks for your response. Forgive me for my noobishness, but which logs?

[joem]

Thanks for your response. Forgive me for my noobishness, but which logs?

Code: Select all

nano  $VESTA/data/templates/web/nginx/default.tpl

Under

Code: Select all

 error_log  /var/log/%web_system%/domains/%domain%.error.log error;

    location / {
        proxy_pass      http://%ip%:%web_port%;

Add

Code: Select all

proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;

If that doesnt work you can also try this,

Right below

Code: Select all

        location ~* ^.+\.(%proxy_extentions%)$ {

Add

Code: Select all

            proxy_cache    off;

Make sure after you save all your changes you rebuild the users web you can do this by,
Login To Vesta CP -> Click Users Tab -> Toggle All -> In the drop down menu select rebuild web

Also restart nginx

Code: Select all

service nginx restart

[Galaxian]

Thanks for your response. Forgive me for my noobishness, but which logs?

Code: Select all

nano  $VESTA/data/templates/web/nginx/default.tpl

Under

Code: Select all

 error_log  /var/log/%web_system%/domains/%domain%.error.log error;

    location / {
        proxy_pass      http://%ip%:%web_port%;

Add

Code: Select all

proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;

If that doesnt work you can also try this,

Right below

Code: Select all

        location ~* ^.+\.(%proxy_extentions%)$ {

Add

Code: Select all

            proxy_cache    off;

Make sure after you save all your changes you rebuild the users web you can do this by,
Login To Vesta CP -> Click Users Tab -> Toggle All -> In the drop down menu select rebuild web

Also restart nginx

Code: Select all

service nginx restart

I've done this, but I have no idea what I've just done or how it pertains to my question regarding error logs. Could you provide an explanation?

[skurudo]

/var/log/apache2 or httpd
this logs