Stopping Bad Bots

Overview

This page describes our strategy for throttling or blocking bots from accessing web sites on our servers.

Bots are programs making web site requests that are not directly driven by a human.

There is some gray area. For example, if a user posts a link on Bluesky, a Bluesky bot will lookup the web page and retrieve the name of the page and possibly an image from the page. That’s technically a bot, but it’s one responding directly to a human’s request.

Also, Stripe, PayPal and Authorize, among many companies, use bots to notify CRM databases that a recurring payment has been processed. These are 100% bots, but ones that should never be throttled or blocked.

Logging

Our bot detection system takes advantage of unused Nginx http response codes to indicate when a bot has been stopped and why. All bots that we stop will have one of these error codes:

  • 480: The user agent is included in the Nginx Ultimate Bad Bot Blocker list.
  • 481: The request referer is included in the Nginx Ultimate Bad Bot Blocker list.
  • 482: The IP is included in the Nginx Ultimate Bad Bot Blocker list because it is trying to masquerade as a Google Bot.
  • 483: The IP is listed in the Spamhaus do not route or peer list.
  • 484: The IP has been banned by Bruce Banner.
  • 485: The IP is in the May First custom list of banned IP ranges (see our inventory hosts.yml file).
  • 486: Based on an analsys of our logs, the IP has triggered a disproportionate number of 404 file not found errors. This category may expand to conver additional suspicious behavior based on our logs.
  • 429: This is the built-in nginx code for an IP that has been rate limited. You should see the reason for the rate limit in the corresponding error log.

General architecture

Our web proxy servers use several files to manage bots:

  1. conf.d/bot-ban-maps.conf: This file is dynamically created on our monitor server. It collects known bot IPs, referers and user agents from a variety of sources and builds out server map variables that can then be used to block requests that match.

  2. conf.d/bot-rate-limit-maps.conf: This file is based on our own settings (in our inventory/hosts.yml file) that define bots that we rate limit based on user agent string (or lack of user agent string). The maps and the rate limits are set in this file but not enforced.

  3. snippets/bad-bots.conf: This file is included in all sites and is in charge of acting on the two mapping files. This file sets the http response code if a request matches. This file also has the limit_req statements that enforce the rate limits set in conf.d/rate-limit-bots.conf file.

Building the configurations

On our monitor server, we have a user botdetector. This user generates a series of nginx configuration files in /var/lib/botdetector/www once a day using the mf-nginx-generate-bad-bot-maps.

All of our proxy web servers, login in via ssh and download these configuration files once a day (using the botdetector-update.service).

Additionally, the botdetector user runs the mf-elastic-generate-bot-stats script to analzye elastic search traffic and score IP addresses based on their likelihood of being a bot. This analysis is stored in a sqlite file (/var/lib/botdetector/bots.db) which can be queries to find out the score of a given IP, e.g.:

sqlite3 ~/bots.db "SELECT * FROM bots WHERE ip = '1.2.3.4'"

The database keeps a history - the nginx configuration files only use the most recent entry.

Maintenance

On monitor, there is a script (/root/elastic-scratch/elastic-web-unique-user-agents) that will generate the most frequently seen user agents. Any user agent seen more than 1,000 times (that is clearly a bot) should be added to the good bots variable in the hosts.yml file.

Challenge

In addition, our web proxy servers can be configured to issue a challenge to any visitor to the site (or any visitor to a location within a site).

The sledge hammer

Sometimes it is just bleak. For those days, we have mf-ip-ban - an IP banning tool that can either ban an IP/IP Range or an entire ASN if that’s what it takes.

mf-ip-ban -h

To get the full usage.

If a web site is getting pounded:

  1. Install asncounter on your computer (it requires the FTP port to be open so won’t work on our servers).
  2. Download the logs to your computer.
  3. Determine which ASNs are the most responsible: awk '{print $1}' /path/to/logs | asncounter