Cheeky and stupid bot …

The bot with user agent Linguee Bot (http://www.linguee.com/bot; bot@linguee.com) just downloaded approximately 2 GiB of stuff from a ViewVC-listing on one of my servers. Just because it’s stupid, but also because it’s so cheeky to ignore robots.txt. So be it. I can indeed match the user agent string … from now on it receives HTTP status code 403.

// Oliver

This entry was posted in Administration, EN, Linux, Opinion, Software, Unix and unixoid and tagged . Bookmark the permalink.

One Response to Cheeky and stupid bot …

  1. anonymous says:

    I was trying to hg clone https://vcs.assarbad.net/ddkbuild/ or https://vcs.assarbad.net/ddkwizard/, but keep getting “abort: HTTP Error 400: Bad Request”. Have no problem accessing them with the web interface, but there is no way to download files there.

Leave a Reply

Your email address will not be published. Required fields are marked *