Bot Challenge Screen

Overview

When a site is being hit with bot traffic that uses a user agent string typically reserved for humans, we can route its traffic through anubis, which poses a challenge to the user in an attempt to determine if it’s a human or not.

How to turn it on

1. Determine which web proxy is being used by the host via DNS lookup.
2. Edit the /etc/nginx/sites-enabled/siteNNNNN.conf file (see below for details)
3. When tested, edit the web conf in the control panel, set Cache Type to “Custom cache settings” and copy the nginx config contents to the Cache Settings field.

Create a new upstream

Add a new upstream to the top of the nginx configuration. Replace the “nnnnn” with the correct site id and the “xxx” with the correct weborigin number.

upstream nnnnn_anubis_weboriginxxx_tls {
  server 127.0.0.1:8xxx max_conns=24;
  zone nnnnn_upstream 32k;
}

Route all traffic

To route all site traffic through anubis, everywhere it has “proxy_pass”, it should read

proxy_pass http://nnnnn_anubis_weboriginxxx_tls;

Replace the “nnnnn” with the correct site id and the “xxx” with the correct weborigin number.

Important: be sure to change https to http when using anubis!

Only route some locations

If you only want to route some locations, you will have a problem, because anubis expects to have access to routes that start with /.within.website (that is how it serves it’s images, css and javascript).

You can resolve this problem by placing this route at the top of your configuration:

 location ~* ^/.within.website {
    proxy_pass http://nnnnn_anubis_weboriginxxx_tls;
  }

Loading the configuration

Once the configuration is changed, run “mf-nginx-t” in order to test the syntax of the configuration file. Note this is a custom script meant to bypass default restrictions in nginx to better simulate May First’s environment.

Once that runs without any errors, run “systemctl reload nginx” to load the new configuration.

Last step

Be sure to edit the control panel, turn on custom caching settings, and copy and paste the final nginx configuration so these changes are saved.

Maintenance

anubis is not in Debian. However, a .deb package is provided via github. We can upgrade by changing the nginx_anubis_version variable in ansible. A debian package seems to be in the works.

We run a separate instance of anubis for each one of our weborigin backends on each webproxy server, each instance is controlled by a systemd unit file and is configured via a .env file in the /etc/anubis directory.