BadBehavior WhiteList For Good Bots & Crawlers

If you use BadBehavior and you don’t scour your BadBehavior logs every day,  you may not realize that “out of the box” BadBehavior may be blocking a few good bots.  Blocking these ‘good’ bots could adversely affect your search rankings, result in poorly targeted contextual ads, and hair-loss.  Lucky for you though, I have no life and scour my logs often, and have put together a list of friendly bots that you probably do not want Bad Behavior to block.

To add these IP’s and CIDRs to your BadBehavior white list, first make a backup copy of whitelist.inc.php.  Then, in the whitelist.inc.php file, find the spot just below where it says:

// DANGER! DANGER! DANGER! DANGER! DANGER! DANGER! DANGER! DANGER!
    // Inappropriate whitelisting WILL expose you to spam, or cause Bad
    // Behavior to stop functioning entirely!  DO NOT WHITELIST unless you
    // are 100% CERTAIN that you should.
    // IP address ranges use the CIDR format.
    // Includes four examples of whitelisting by IP address and netblock.
    $bb2_whitelist_ip_ranges = array(

Depending on what version of Bad-Behavior you are running, you will probably already see a few address there, so just below those, add the IPs from my list below.  Be sure to put double-quotes around each IP/CIDR and don’t forget the comma after each one.  And of course, just like the warning says: do not WHITELIST unless you are 100% CERTAIN that you should.. Don’t take some lame-o blogger’s word for it!  Do a quick WHOIS lookup of each one before adding.

These are the bots that I have added to my whitelist.inc.php file which were being blocked by Bad Behavior by default:

"65.52.0.0/14",       //Microsoft crawlerz
"66.228.160.0/19",    //Overture crawlers
"66.249.64.0/19",     // big G
"74.6.0.0/16",        //yahoo
"209.191.64.0/18",    //Yahoo
"216.145.48.0/20",    //Yahoo
"216.252.116.53",     //Yahoo
"219.142.53.19",     //microsoft china (this one is questionable)

After adding the IP address / CIDRs, save the whitelist.inc.php file back to your server and the changes will take effect immediately.

Be sure to test the changes by viewing both a cached and uncached page of your site (For Drupal this means logged-in and not logged-in)!  Any typo, like a missing double-quote or missing comma will break your site!

These good bots will now be able to roam your site freely and with reckless abondon..

If you have more to add, or if you don’t think adding the ones above are a good idea, please post a comment ’cause lord knows half the time I don’t know WTF I’m doing.

Leave a Reply

Your email address will not be published.